Methods and Software for Screening and Diagnosing Skin Lesions and Plant Diseases

ABSTRACT

Provided herein are portable imaging systems, for example, a digital processor-implemented system for the identification and/or classification of an object of interest on a body, such as a human or plant body. The systems comprise a hand-held imaging device, such as a smart device, and a library of algorithms or modules that can be implemented thereon to process the imaged object, extract representative features therefrom and classify the object based on the representative features. Also provided are methods for the identifying or classifying an object of interest on a body that utilize the algorithms and an automated portable system configured to implement the same.

CROSS-REFERENCE TO RELATED APPLICATIONS

This non-provisional application claims benefit of priority under 35U.S.C. §119(e) of provisional application U.S. Ser. No. 61/616,633,filed Mar. 28, 2012, now abandoned, the entirety of which is herebyincorporated by reference.

FEDERAL FUNDING LEGEND

This invention was made with governmental support under Grant NumberIR21AR057921 awarded by the National Institutes of Health. Thegovernment has certain rights in the invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the fields of dermoscopy and softwarefor the practice thereof on handheld biomedical screening devices.Specifically, the present invention provides a library of imageprocessing and texture analysis algorithms that run on embedded devicesfor screening lesions and for plant diseases.

2. Description of the Related Art

The American Cancer Society predicts that there will be approximately68,130 new cases of melanoma and 8,700 deaths because of melanoma in USin 2010 (1). Thus, detection of an early melanoma is of paramountimportance for successful skin cancer screening. The use of dermoscopy,an imaging technique to visualize structures inside pigmented skinlesions beyond the naked eye, and computerized systems for automatedclassification of dermoscopic images (2) can drastically improve thediagnostic accuracy of early melanoma. Image classification usinginterest points has shown success in previous studies (3).

In recent years, cellular phones have made the transition from simplededicated telephony devices to being small, portable computers with thecapability to perform complex, memory- and processor-intensiveoperations. These new smart devices, generally referred to assmartphones, provide the user with a wide array of communication andentertainment options that until recently required many independentdevices. Given advances in medical imaging, smartphones provide anattractive vehicle for delivering image-based diagnostic services at alow cost.

As such, the new generation of smart handheld devices with sophisticatedhardware and operating systems has provided a portable platform forrunning medical diagnostic software, such as the heart rate monitoring(4), diabetes monitoring (5), and experience sampling (6) applications,which combine the usefulness of medical diagnosis with the convenienceof a handheld device. Their light operating systems, such as the Apple®iOS® and Google® Android®, the support for user friendly touch gestures,the availability of an SDK for fast application development, the rapidand regular improvements in hardware, and the availability of fastwireless networking over Wi-Fi and 3G make these devices ideal formedical applications.

Thus, there is a recognized need in the art for algorithms for improveddetection, analysis and classification of skin lesions that can run ondevices with limited memory and computational speed. More specifically,the prior art is deficient in image sampling, processing and textureanalysis methods, algorithms and plug-in features, for detection anddiagnosis of skin and ocular diseases and plant diseases, that areconfigured to operate on smart handheld devices. The present inventionfulfills this long-standing need and desire in the art.

SUMMARY OF THE INVENTION

The present invention is directed to a portable imaging system. Theportable imaging system comprises a hand-holdable imaging device havinga digital camera, a display, a memory, a processor and a networkconnection and a library of algorithms tangibly stored in the memory andexecutable by the processor, where the algorithms are configured foridentification of an object of interest present on a body. The presentinvention is directed a related portable imaging system furthercomprising algorithms tangibly stored and processor executablealgorithms configured to display the object of interest and results ofthe classification thereof.

The present invention is directed to a method for identifying an objectof interest present on a body. The method comprises acquiring an imageof the object of interest on the body via the imaging device comprisingthe portable imaging system described herein and processing the acquiredobject image via the algorithms tangibly stored in the imaging device.The object in the image is identified based on patterns of featurespresent in the imaged object, thereby identifying the object of intereston the body. The present invention is directed to a related methodfurther comprising the step of displaying the results of imageprocessing as each result occurs.

The present invention is directed further to a digitalprocessor-implemented system for classifying an object of interest on ananimal or plant body in real time. The system comprises a portable smartdevice comprising the processor, a memory and a network connection andmodules tangibly stored in the memory. The modules comprise a module forsegmentation of an imaged object, a module for feature extraction withinthe segmented object image and a module for classification of the objectbased on extracted features. The present invention is directed to arelated digital processor-implemented system further comprising a moduletangibly stored in the memory for display of the object of interest andresults of the classification thereof.

The present invention is directed further still to a digitalprocessor-implemented method for classifying an object of interest on ananimal or plant body in real time. The method comprises the processorexecutable steps of digitally imaging the object of interest with thesmart device comprising the digital processor-implemented systemdescribed herein, processing the digital image through the systemmodules and displaying the processed images and classification resultson a display comprising the smart device. The modules comprisealgorithms configured for segmenting the image based on saliency valuesto identify pixels thereof as comprising the imaged object or thebackground of the image to obtain an object boundary, extractingfeatures from regions within the object boundary and comparing theextracted features to known object features in a support vector machinetrained on the known features to obtain a classification of the object.

The present invention is directed further still to a digitalprocessor-readable medium tangibly storing processor-executableinstructions to perform the digital processor implemented methoddescribed herein.

The present invention is directed further still to a computer-readablemedium tangibly storing a library of algorithms to classify an object ofinterest on a human or plant body. The algorithms comprisesprocessor-executable instructions operable to obtain luminance and colorcomponents of the imaged object, classify pixels comprising the image asobject pixels, if they belong to a common luminance and colorforeground, as background pixels if they belong to a common luminanceand color background or as remaining pixels, apply a classifier to theremaining pixels to classify them as object or foreground, calculate asaliency value for a plurality of patches within the segmented objectand separate the patches into regions based on the saliency values,calculate an average intensity for the regions to identify them as ahigher or as a lower intensity region, determine a sampling percentagefor the two intensity regions, sample patches within the intensityregions by corresponding sampling percentages, extract one or morefeature representations for the object, train a support vector machine(SVM) with known manually segmented objects, and classify the objectbased on the extracted feature representations inputted into the SVM.

Other and further aspects, features and advantages of the presentinvention will be apparent from the following description of thepresently preferred embodiments of the invention given for the purposeof disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the matter in which the above-recited features, advantages andobjects of the invention, as well as others which will become clear areattained and can be understood in detail, more particular descriptionsand certain embodiments of the invention briefly summarized above areillustrated in the appended drawings. These drawings form a part of thespecification. It is to be noted, however, that the appended drawingsillustrate preferred embodiments of the invention and, therefore, arenot to be considered limiting in their scope.

FIGS. 1A-1C depict segmentation examples based on saliency (FIG. 1A) andnonuniform (FIG. 1B) sampling and classification performance fordifferent ratios of sampling densities between the informative andhomogenous region (FIG. 1C). Blue circles represent patch centers in themore informative (darker) region, while red crosses correspond to patchcenters in the less informative (more homogeneous) region.

FIGS. 2A-2B depict an original prior art image (7) (FIG. 2A) andinterest points (FIG. 2B) detected by SIFT using a threshold adjusted toretain 0.25% of points in the lesion.

FIGS. 3A-3B depict an original prior art image (7) (FIG. 3A) and Frangifiltered image (FIG. 3B). Higher responses, i.e., brighter spots, areseen at curvilinear structures in the periphery of the lesion.

FIG. 4 depicts the segmentation inside the lesion in FIG. 2B.

FIGS. 5A-5B illustrate the effect of the total points sampled onclassification accuracy for balanced accuracy (BAC) (FIG. 5A) and areaunder the receiver operating characteristic curve (AUC) (FIG. 5B) for apigmented skin lesion (PSL).

FIGS. 6A-6B are comparisons of various sampling schemes for 24×24patches for balanced accuracy (BAC) (FIG. 6A) and area under thereceiver operating characteristic curve (AUC) (FIG. 6B).

FIGS. 7A-7B are comparisons of single scale and multi-scale samplingsfor various sampling schemes for balanced accuracy (BAC) (FIG. 7A) andarea under the receiver operating characteristic curve (AUC) (FIG. 7B).

FIG. 8 depicts the lesion classification process with steps selecting apatch from the lesion, applying a 3-level Haar wavelet transform,extracting texture features, building a histogram using the clustercenters obtained during training, and inputting the histogram to thetrained SVM classifier to classify the lesion.

FIGS. 9A-9D depicts the scanning application on an Apple® iPhone®device.

FIGS. 10A-10C illustrate the error ratio distribution of segmentationmethods Fuzzy-C Means (FIG. 10A), ISODATA (FIG. 10B) and Active Contour(FIG. 10C) on the dataset of 1300 skin lesion images. The dotted linemarks the threshold for correct segmentation.

FIG. 11 depicts the ROC curve for lesion classification.

FIG. 12 depicts the results of blue-whitish veil detection on skinlesion images.

FIGS. 13A-13B depict varying patch size (FIG. 13A) and varying local binsize (FIG. 13B). Sen1 and Spec1 represent sensitivity and specificity offirst global approach. Sen2 and Spec2 represent sensitivity andspecificity of second global approach.

FIG. 14 depicts varying global bin size. Blue bar represents sensitivityand red bar represents specificity.

FIGS. 15A-15B depict scaling light intensity (FIG. 15A) and shiftinglight intensity (FIG. 15B). Sen1 and Spec1 represent sensitivity andspecificity of first global approach. Sen2 and Spec2 representsensitivity and specificity of second global approach.

FIGS. 16A-16B depict the smartphone screen showing the image with menuchoices (FIG. 16A) and the results and diagnosis after comparison withthe 7-Points Criteria for detecting melanoma (FIG. 16B).

FIGS. 17A-17B depict a flowchart of the classification process in eachof the algorithmic segmentation (FIG. 17A) and feature extraction andclassification (FIG. 17B) modules.

18 FIGS. 18A-18F are examples of Buruli ulcer (BU) segmentation showingthe original BU image with manual segmentation (FIG. 18A), the initialmask by fusion (FIG. 18B), level set segmentation in color (FIG. 18C)and luminance (FIG. 18D) channels, segmentation after pixelclassification (FIG. 18E), and the final segmentation result (FIG. 18F).The line(s) around the lesions show the result for automaticsegmentation (FIGS. 18B-18E), the ground true from an expertdermatologist (FIG. 18A) or both (FIG. 18F).

19 FIGS. 19A-19D depict different segmentation methods by AT (FIG. 19A),GVF (FIG. 19B), LS (FIG. 19C), and the instant segmentation method (FIG.19D) where results from automatic segmentation and ground true from anexpert dermatologist are included.

FIGS. 20A-20B are examples of image patterns closest to the two clustercentroids for Buruli lesions (FIG. 20A) and non-Buruli lesions (FIG.20B).

FIGS. 21A-21B illustrate the effect of sampling strategy onclassification performance. FIG. 21A shows the effect of patch numberand FIG. 21B shows the effect of patch size.

FIGS. 22A-22B illustrates ulcer variation of a Buruli ulcer from earlyto late stage (FIG. 22A) and demonstrates that the algorithms candistinguish between early and late stage ulcers (FIG. 22B).

FIG. 23 depicts the architecture for the multispectral imaging process.

FIG. 24 depicts a flowchart of the classification process in algorithmicfeature extraction and classification modules for multispectral images.

FIG. 25 depicts a flowchart of the classification process in algorithmicfeature extraction and classification modules for an optical skin model.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, the term “a” or “an”, when used in conjunction with theterm “comprising” in the claims and/or the specification, may refer to“one,” but it is also consistent with the meaning of “one or more,” “atleast one,” and “one or more than one.” Some embodiments of theinvention may consist of or consist essentially of one or more elements,method steps, and/or methods of the invention. It is contemplated thatany method or composition described herein can be implemented withrespect to any other method or composition described herein.

As used herein, the term “or” in the claims refers to “and/or” unlessexplicitly indicated to refer to alternatives only or the alternativesare mutually exclusive, although the disclosure supports a definitionthat refers to only alternatives and “and/or.”

As used herein, the term “about” refers to a numeric value, including,for example, whole numbers, fractions, and percentages, whether or notexplicitly indicated. The term “about” generally refers to a range ofnumerical values (e.g., +/−5-10% of the recited value) that one ofordinary skill in the art would consider equivalent to the recited value(e.g., having the same function or result). In some instances, the term“about” may include numerical values that are rounded to the nearestsignificant figure.

As used herein, the terms “body” and “subject” refer to a mammal,preferably human, or to a plant.

As used herein, the term “object” in reference to a body or a subject,refers to a lesion, wound, ulcer or other condition, or the skin or aregion comprising the same that is located on the subject or body orrefers to an area on the subject or body suspected of being or ismalignant or associated with a disease or other pathophysiologicalcondition.

In one embodiment of the present invention there is provided a portableimaging system, comprising a hand-holdable imaging device having adigital camera, a display, a memory, a processor and a networkconnection; and a library of algorithms tangibly stored in the memoryand executable by the processor, said algorithms configured foridentification of an object of interest present on a body. Further tothis embodiment the portable imaging system comprises algorithmstangibly stored and processor executable algorithms configured todisplay the object of interest and results of the classificationthereof.

In both embodiments the algorithms may comprise processor-executableinstructions to segment the imaged object to detect a border of theobject; extract features from the segmented object image; and classifythe object based on the extracted features.

In an aspect of both embodiments the processor-executable instructionsto segment the object may function to determine an initial contour ofthe imaged object; classify pixels as contained within the initialcontour as foreground, as contained without the initial contour asbackground or as remaining pixels; and apply a classifier to theremaining pixels for classification as foreground or background.

In another aspect of both embodiments the processor-executableinstructions to extract features may function to divide the segmentedobject image into regions based on saliency values calculated for atleast one patch within the segmented object; divide the regions into tworegions of higher or lower intensity based on average intensity valuesthereof; and extract feature representations from a sampling of patcheswithin the intensity regions based on sampling percentages determinedfor the regions.

In yet another aspect of both embodiments the processor-executableinstructions to classify the object may function to input the extractedfeature representations into a support vector machine trained withmanually segmented objects; and classify the object based on acomparison of the inputted extracted features with those in the trainedsupport vector machine.

In all embodiments and aspects thereof the hand-held imaging device maybe a smart device. Also, the body may be a human body or a plant body.In addition representative examples of the object of interest are alesion, an ulcer, or a wound.

In another embodiment of the present invention there is provided amethod for identifying an object of interest present on a body,comprising acquiring an image of the object of interest on the body viathe imaging device comprising the portable imaging system describedsupra; processing the acquired object image via the algorithms tangiblystored in the imaging device; and identifying the object in the imagebased on patterns of features present in the imaged object, therebyidentifying the object of interest on the body.

Further to this embodiment the method comprises displaying the resultsof image processing as each result occurs. In both embodimentsidentifying the object of interest occurs in real time. Also, in bothembodiments the object of interest may be a melanoma or a Buruli ulcer.

In yet another embodiment of the present invention there is provided adigital processor-implemented system for classifying an object ofinterest on an animal or plant body in real time, comprising a portablesmart device comprising the processor, a memory and a networkconnection; and modules tangibly stored in the memory comprising amodule for segmentation of an imaged object; a module for featureextraction within the segmented object image; and a module forclassification of the object based on extracted features. Further tothis embodiment the digital processor-implemented system comprises amodule tangibly stored in the memory for display of the object ofinterest and results of the classification thereof. Representativeexamples of the object may be a lesion, an ulcer or a wound.

In both embodiments the segmentation module comprises processorexecutable instructions to obtain luminance and color components of theimaged object; classify pixels comprising the image as object pixels, ifthey belong to a common luminance and color foreground, as backgroundpixels if they belong to a common luminance and color background or asremaining pixels; and apply a classifier to the remaining pixels toclassify them as object or foreground.

In an aspect of both embodiments the feature extraction module maycomprise processor executable instructions to calculate a saliency valuefor a plurality of patches within the segmented object and separate thepatches into regions based on the saliency values; calculate an averageintensity for the regions to identify them as a higher or as a lowerintensity region; determine a sampling percentage for the two intensityregions; sample patches within the intensity regions by correspondingsampling percentages; and extract one or more feature representationsfor the object. Also in both embodiments the classification modulecomprises processor executable instructions to train a support vectormachine (SVM) with known manually segmented objects; and classify theobject based on the extracted feature representations inputted into theSVM.

In another aspect the feature extraction module may comprise processorexecutable instructions to read input white light image as RGB and thesegmentation result of the region; read input multispectral images incolor channels and transform to gray scale; register multispectralimages via maximization of mutual information with white light image asreference; extract feature representations within the ROI ofmultispectral images and within white light images; and select one ormore relevant features from a pool of the extracted features.

In yet another aspect the feature extraction module may compriseprocessor executable instructions to read input white light image as RGBand the segmentation result of the region; read input multispectralimages in color channels and transform to gray scale; registermultispectral images via maximization of mutual information with whitelight image as reference; determine V_(mel), V_(blood), and V_(oxy) foreach ROI pixel to reconstruct maps of melanin, blood and oxygenatingpercentage; extract feature representations within the ROI from thereconstructed maps; and select one or more relevant features from a poolof the extracted features.

In yet another embodiment of the present invention there is provided adigital processor-implemented method for classifying an object ofinterest on an animal or plant body in real time, comprising theprocessor executable steps of digitally imaging the object of interestwith the smart device comprising the digital processor-implementedsystem described supra; processing the digital image through the systemmodules, the modules comprising algorithms configured for segmenting theimage based on saliency values to identify pixels thereof as comprisingthe imaged object or the background of the image to obtain an objectboundary; extracting features from regions within the object boundary;and comparing the extracted features to known object features in asupport vector machine trained on the known features to obtain aclassification of the object; and displaying the processed images andclassification results on a display comprising the smart device. In thisembodiment the support vector machine may be trained on featurescomprising a melanoma or a Buruli ulcer.

In a related embodiment there is provided a digital processor-readablemedium tangibly storing processor-executable instructions to perform thedigital processor implemented method described herein.

In yet another embodiment of the present invention there is provided acomputer-readable medium tangibly storing a library of algorithms toclassify an object of interest on a human or plant body, said algorithmscomprising processor-executable instructions operable to obtainluminance and color components of the imaged object; classify pixelscomprising the image as object pixels, if they belong to a commonluminance and color foreground, as background pixels if they belong to acommon luminance and color background or as remaining pixels; apply aclassifier to the remaining pixels to classify them as object orforeground; calculate a saliency value for a plurality of patches withinthe segmented object and separate the patches into regions based on thesaliency values; calculate an average intensity for the regions toidentify them as a higher or as a lower intensity region; determine asampling percentage for the two intensity regions; sample patches withinthe intensity regions by corresponding sampling percentages; extract oneor more feature representations for the object; train a support vectormachine (SVM) with known manually segmented objects; and classify theobject based on the extracted feature representations inputted into theSVM.

In one aspect the instructions to extract one or more featurerepresentations for the object may calculate a saliency value for aplurality of patches within the segmented object and separate thepatches into regions based on the saliency values; calculate an averageintensity for the regions to identify them as a higher or as a lowerintensity region; determine a sampling percentage for the intensityregions; sample patches within the intensity regions by correspondingsampling percentages; and extract the one or more featurerepresentations for the object.

In another aspect the instructions to extract one or more featurerepresentations for the object may read input white light image as RGBand the segmentation result of the region; read input multispectralimages in color channels and transform to gray scale; registermultispectral images via maximization of mutual information with whitelight image as reference; extract feature representations within the ROIof multispectral images and within white light images; and select one ormore relevant features from a pool of the extracted features.

In yet another aspect the instructions to extract one or more featurerepresentations for the object may read input white light image as RGBand the segmentation result of the region; read input multispectralimages in color channels and transform to gray scale; registermultispectral images via maximization of mutual information with whitelight image as reference; determine V_(mel), V_(blood), and V_(oxy) foreach ROI pixel to reconstruct maps of melanin, blood and oxygenatingpercentage; extract feature representations within the ROI from thereconstructed maps; and select one or more relevant features from a poolof the extracted features.

The present invention provides a library of algorithms, algorithmmodules, plug-ins, and methods utilizable on a handheld portable imagingdevice, such as a smart device, for the identification and/orclassification of an object of interest on a body. The library may be aC/C++ based library and may be downloaded and accessed as a plug-in orstored in the memory on a handheld smart device. Representative examplesof such smart devices are, but not limited to, Apple® iOS®-baseddevices, such as iPhone®, iPad® and iPod Touch®, which are trademarks ofApple Inc., registered in the U.S. and other countries, and Android®based devices, which is a registered trademark of Google Inc. Thelibrary comprises algorithms useful for the processing, texture analysisand classification of an image of an object of interest on a human orplant body skin lesion as malignant or benign. The algorithms aregenerally applicable to segmentation, feature extraction andclassification of the object. For example, the object of interest may beidentified or classified as a benign or malignant lesion, an ulcer, forexample, but not limited to, a Buruli ulcer, or a wound.

Thus, provided herein are methods, particularly, digitalprocessor-implemented methods for identifying or classifying the objectof interest via the algorithms or modules comprising the same. Themethod utilizes the algorithmic library implemented on a handheld smartdevice, for example, a smartphone, as described herein. An image isacquired with the smartphone. Image acquisition optionally may utilizean external attachment that can provide illumination and magnification.The library of algorithms or modules is implemented on the smart deviceto operate on the image and a classification score is obtained, asdescribed. As such cancers and other diseases of a human or plant bodymay be diagnosed. Particularly, the method enables diagnosis of a skincancer, such as melanoma, or a skin disease, such as Buruli ulcer. Asufficient resolution enables the detection of and distinction amongother types of lesions, ulcers or wounds. Moreover, the algorithms areconfigured to enable to run a Buruli analysis and a skin lesion analysisat the same time. The algorithms provided herein also are configured toprocess multispectrally-acquired images or optical images obtained withlights of different frequencies utilizing feature extraction andclassification algorithms.

The present invention also includes a digital processor-readable orcomputer-readable medium that can tangibly store the algorithms and/orthe processor-executable instructions or methods contained therein. Suchreadable media are well-known and standard in the art and can comprise amemory, such as on the smart device or other networked computer or adiskette, memory stick or flash drive from which the algorithms can bedownloaded to the smart device, for example as a plug-in.

Moreover, the present invention provides an automated portable systemfor identification or classification of the object of interest. Thesystem comprises a smart handheld device having an operating system,memory, processor, display screen, and lens and means for imageacquisition, such as a digital camera, and is networkable, as iswell-known and standard in the art. The system is configured to acquireand display an image of the object and to operate the library/modules ofalgorithms on the acquired image to process, identify patterns of objectfeatures, and classify the object. While the system is configured tooperate on the smart handheld device, the device can wirelessly port theinformation or results to another smart device or desktop computer bymethods well-known in the art.

It is contemplated that the library of algorithms provided herein may beutilized to detect ocular diseases, such as glaucoma. With an imagingtechnology suitable to allow visualization of the retinal tissue at anadequate resolution, the library of algorithms provided herein may beutilized for automatic analysis of the acquired image noninvasively inreal time. Moreover, a handheld smart device comprising the library maybe useful in obtaining and analyzing in real time infrared images ofleaves to detect plant diseases or insect infestations or fungalinfections in good time to save the crop. This results in successfulagricultural management and improved crop productivity.

The following examples are given for the purpose of illustrating variousembodiments of the invention and are not meant to limit the presentinvention in any fashion.

Example 1 Methods

An image, such as skin lesion, can be acquired using a smartphonecamera, with or without an external attachment that can provideillumination and magnification, or can be loaded from the photo libraryto provide the diagnosis in real time. The application can processimages taken either with the iPhone® camera or taken with an externalcamera and uploaded to the image library on the phone. To test theapplication, images from a large commercial library of skin cancerimages that were annotated by dermatologists (7) are uploaded to thephone. Intra-observer and inter-observer agreement could be low forcertain criteria (8). All images were segmented manually to provide anevaluative level against which the automated techniques of theapplications presented herein were compared. Prior to processing, ifnecessary, the images are converted from color to 256 level greyscale.

The iPhone 4® is a smartphone developed by Apple Inc., and features anApple® A4 ARM® processor running at 1 GHz, 512 MB DDR SDRAM (AdvancedRISC Machines, Ltd), up to 32 GB of solid state memory, a 5 MPixelbuilt-in camera with LED flash, and 3G/Wi-Fi/Bluetooth® (Bluetooth SIG)communication networks. Thus, it has all the features, computationalpower, and memory needed to run the complete image acquisition andanalysis procedure in quasi real time. There is lot of potential forsuch a device in medical applications, because of its low cost,portability, ease of use, and ubiquitous connectivity.

7-Point Checklist Criteria for Diagnosis of Melanoma

The 7-point checklist includes seven dermoscopic features that can bedetected with high sensitivity and decent specificity by even lessexperienced clinicians. The seven points of the list are subdivided intothree major and four minor criteria, reflecting their importance indefining a melanoma. To score a lesion, the presence of a majorcriterion is given two points and that of a minor one point. If thetotal score is greater than or equal to 3, the lesion is classified asmelanoma.

The major criteria are: 1) Atypical pigment network: Black, brown, orgray network with irregular meshes and thick lines; 2) Blue-whitishveil: Confluent gray-blue to whitish-blue diffuse pigmentationassociated with pigment network alterations, dots/globules and/orstreaks; and 3) Atypical vascular pattern: Linear irregular or dottedvessels not clearly combined with regression structures and associatedwith pigment network alterations, dots/globules and/or streaks.

The minor criteria are 1) Irregular streaks: Irregular more or lessconfluent, linear structures not clearly combined with pigment networklines; 2) Irregular pigmentation: Black, brown, and/or gray pigmentareas with irregular shape and/or distribution; 3) Irregulardots/globules: Black, brown, and/or gray round to oval, variously sizedstructures irregularly distributed; 4) Regression structures: Associatedwhite scar-like and gray blue, peppering, multiple blue gray dots.

Example 2 Non-Uniform Sampling for Bag-of-Features ClassificationDataset

A dataset with 645 epiluminescence microscopy (ELM) images, in which 491lesions are benign and 154 lesions are melanoma, was used. The imagesare mainly collected from a commercial database (7). The total imagesize ranges from 712×454 to 1,024×768 pixels, while the lesion sizeranges from 7,662 to 804,527 pixels.

Procedure

For bag-of-features-based classification, first, patches are extractedfrom every lesion, and then, for each patch, patch descriptors aregenerated using color moments and Haar wavelet coefficients, whichcapture the color and texture information. Then, each patch is assignedto a codeword from a pre-learned codebook, using hard assignment, asdescribed herein. After that, a final feature vector is generated bypooling the assignments of all patches extracted from the lesion. Forthe classifier, support vector machines (SVMs) with a χ² kernel areused, which currently represent state-of-art settings for abag-of-features model.

Ten times ten-fold stratified cross-validation was performed to evaluatethe performance of the method. Performance criteria include sensitivity,specificity, balanced accuracy (BAC, i.e., average of sensitivity andspecificity), and area under the receiver operating characteristic curve(AUC). For sensitivity and specificity, both the mean and 95% confidenceinterval estimated from a binomial distribution are reported, and theaverage for BAC. Similarly, for AUC both the mean value and the standarddeviation obtained are shown.

Occurrence-Based Contextual Saliency

The saliency measure S_(i) ^(o) uses co-occurrence information betweenpatches and codewords in images. Given a patch x_(i), saliency isdefined as the average likelihood of the image patches,

$\begin{matrix}{\mspace{79mu} {S_{i}^{o} = {\frac{1}{n}\text{?}{{\phi \left( x_{j} \middle| x_{i} \right)}.\text{?}}\text{indicates text missing or illegible when filed}}}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$

Here, φ(x_(j)|x_(i)) can be interpreted as a compatibility score betweenx_(i) and x_(j). That is, if patch x_(i) has been seen in an image, ahigher score for patch x_(j) means that it will appear in the same imagewith a higher probability. The function φ(x_(j)|x_(i)) can be computedin the following steps.

Let x_(i) and y_(i) denote a patch and the corresponding patchdescriptor, respectively. Then each image patch x_(i) is assigned to acodeword w_(a) in the codebook. These assignments may be soft or hard,and the probability of patch being assigned to codeword is given by

α_(ia) =P(w _(a) y _(i))  (Eq. 2).

The value w_(a) is discrete, while the values of α_(ia) for allcodewords sum to one, i.e.,

Σ

^(m) p(w _(a) |y _(i))=1.

The value of φ(x_(j)|x_(i)) is computed by marginalizing over allpossible codeword assignment for x_(j).

$\begin{matrix}{\mspace{79mu} {{{\phi \left( x_{j} \middle| x_{i} \right)} = {\text{?}{p\left( {\left. w_{k} \right|\text{?}} \right)}{\phi \left( {\left. w_{k} \right|\text{?}} \right)}}},{\text{?}\text{indicates text missing or illegible when filed}}}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$

where φ(w_(b)|x_(j)) is the compatibility score between w_(b) and x_(i).In the same way, φ(w_(b)|x_(i)) can be obtained by marginalizing overall possible codeword assignment for x_(i)

$\begin{matrix}{\mspace{79mu} {{\phi \left( w_{k} \middle| x_{i} \right)} = {\text{?}{p\left( \text{?} \right)}{{p\left( \text{?} \right)}.\text{?}}\text{indicates text missing or illegible when filed}}}} & \left( {{Eq}.\mspace{14mu} 4} \right)\end{matrix}$

By rearranging Equation 1, Equation 3, and Equation 4, Eq. 5 wasobtained

$\begin{matrix}{\text{?} = {\frac{1}{n}\text{?}{p\left( \text{?} \right)}{p\left( \text{?} \right)}{{p\left( \text{?} \right)}.\text{?}}\text{indicates text missing or illegible when filed}}} & \left( {{Eq}.\mspace{14mu} 5} \right)\end{matrix}$

where n is the number of pixels, m is the number of clusters,

$\begin{matrix}{{p\left( w_{a} \middle| y_{i} \right)} = \left\{ \begin{matrix}1 & {{{if}\mspace{14mu} y_{i}} \Subset w_{a}} \\0 & {otherwise}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 6} \right) \\{{p\left( w_{b} \middle| y_{j} \right)} = \left\{ \begin{matrix}1 & {{{if}\mspace{14mu} y_{j}} \Subset w_{b}} \\0 & {otherwise}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 7} \right)\end{matrix}$

where p(w_(b)|w_(a)) is the empirical conditional probability ofobserving codeword w_(b) given that codeword w_(a) has been observedsomewhere in the image. These can be learned through Maximum LikelihoodEstimator (MLE) counts from the training images.

Since saliency measures how well an individual patch can predict theoccurrence of other patches in the same image, pixels with highersaliency values always belong to the relatively more homogenousbackground region. Therefore, some informative patches with lowersaliency values are missed when using saliency-based image sampling.

Nonuniform Sampling Strategy

The basic idea of nonuniform sampling is to sample more patches fromdermoscopically interesting regions, which are obtained by segmentationaccording to patch saliency and pixel intensity. It consists in fourmain steps:

1. Calculate saliency values for all patches;

2. Separate the lesion into two regions based on saliency;

3. Choose informative and homogeneous regions according to pixelintensities;

4. Decide sampling densities for the two separate regions.

In Step 1, saliency values are calculated using Equation 3 for eachpatch, and then k-means clustering is applied to separate the lesioninto two regions. Subsequently, the region with lower pixel intensitieswas chosen as the informative region, and the other region as thehomogenous one. That is because with pigmented skin lesions,dermatologists always pay more attention to the dark areas of a lesionto diagnose a melanoma. Then the sampling density for each region waschosen. When random sampling is applied to these two distinct regions,more patches are extracted from the informative region and fewer fromhomogeneous one. The sampling densities are controlled by the followingequation,

P _(i)=(α·A _(i) /α·A _(i) +A _(h))×100%  (Eq. 8)

where P_(i) represents the percentage of patches sampled from theinformative region, and A_(i) and A_(h) are the areas of the informativeand homogeneous regions, respectively. The ratio of sampling densitiesof informative to homogeneous region is α.

The coefficient α can be fixed or be allowed to vary dynamically. Adynamic α can be obtained by considering the saliency values of the tworegions, which means that a big difference in saliency will result in abig difference in sampling density between two regions. FIGS. 1A-1Bshows examples of saliency-based sampling and nonuniform sampling. It isobvious that saliency-based sampling incorrectly captures moreinformation from the homogeneous background and misses informativepatches that it considers as nonsalient. In contrast, the proposednonuniform method correctly emphasizes the region that contains moreDIPs.

Comparison of Sampling Strategies

The instant method is compared against plain saliency-based sampling(8), whereby patches with higher saliency values are considered ofhigher importance. To avoid the influence of the number of patchesextracted and of the patch size, for both sampling strategies the patchsize was fixed to 24×24 pixels and the patch percentage to approximately4%. Table 1 shows the resulting classification performance for the twoapproaches. It can be seen that the proposed method achieves betteraccuracy than saliency based sampling. That is because patches withhigher saliency values always belong to the relatively more homogenousregion of the lesion, while the most informative patches from the skinlesion that are less frequent, and thus have lower saliency values, aremissed. On the other hand, the method uses saliency as a measure toseparate the image into more informative and less informative regions,while pixel intensities are used to identify the informative one. Inthis way, when analyzing pigmented skin lesions, more patches aresampled from the informative region that contains more DIPs.

TABLE 1 Classification performance using different sampling strategiesSensitivity Specificity BAC AUC Methods (95% CI) (95% CI) (95% CI) (std)Saliency 89.53 84.45 88.99 96.78 ± [86.91, 91.79] [85.73, 90.81] [86.32,91.30] 2.12 Nonuniform 93.67 92.00 92.83 98.69 ± [91.50, 95.42] [89.63,93.97] [90.57, 94.70] 1.12

Effects of Saliency

To demonstrate the benefits of using saliency for lesion sampling, as acontrol another sampling method was added, which both segments an imageand chooses the informative region based on pixel intensity. Again,patches of size 24×24 pixels covering 4% of the total lesion area aresampled with this method, and random sampling is applied in the separateregions. Table 2 shows the resulting classification performance for thetwo approaches. It can be seen that lesion separation according to patchsaliency can achieve a better classification performance than separationbased only on pixel intensity. Thus, saliency provides an effective wayto separate a lesion into informative and homogeneous regions.

TABLE 2 Classification performance using different sampling strategiesSensitivity Specificity BAC AUC Methods (95% CI) (95% CI) (95% CI) (std)Intensity 88.67 86.49 87.58 95.89 ± [85.96, 91.01] [83.61, 89.03][84.79, 90.02] 2.45 Nonuniform 93.67 92.00 92.83 98.69 ± [91.50, 95.42][89.63, 93.97] [90.57, 94.70] 1.12

Effects of Sampling Density

The ratio of sampling densities between informative and homogeneousregions can also affect classification accuracy. For the nonuniformsampling method, different values for the coefficient α in Equation 4were tested, which represent different ratios of sampling densitieswhose influence is shown in FIG. 1C. When a equals one, nonuniformsampling is equivalent to uniform sampling. As a increases above one,and more patches are sampled from the informative region, theclassification accuracy also increases. However, when a becomes toolarge, the overall performance decreases as well. This suggests thatthere is a minimum amount of complementary information provided by thehomogeneous region that is essential for accurate classification. Insummary, patches from informative regions that contain moredermoscopically interesting features should be sampled more densely, butpatches from homogeneous regions which provide complementary informationshould not be ignored. The best performance can be achieved when α lieswithin (1, 2).

Example 3 Evaluation of Sampling Strategies of Dermoscopic InterestPoints (DIPs) in Melanomas Dataset

A dataset with 1,505 epiluminescence microscopy (ELM) images, in which1,098 lesions are benign and 407 lesions are melanoma were used. Theimage size ranges from 712×454 to 1,024×768 pixels, and lesion sizeranges from 7,662 to 804,527 pixels. Manual segmentation of all lesionsis used to ensure that evaluation of the various sampling strategies isnot affected by possible differences in automated identification of thelesion boundary.

Procedure

The lesion classification procedure (9) consists of five main steps:image sampling, feature extraction, coding, pooling, and finalclassification (10). For a given image, identify DIPs inside the lesionare identified first and then a patch is extracted around each DIP. Oneach patch, several low level texture and color features were computedusing Haar wavelets and color moments, which are important for melanomadetection. In the coding stage, a patch is assigned to a codeword from apre-learned codebook using hard or soft assignment. Herein, each patchis assigned to its nearest neighbor in the codebook with hardassignment. The assignments of all patches extracted from a lesion arepooled into one feature vector. The last step is to classify the lesionbased on the feature vector obtained from pooling.

Ten times ten-fold stratified cross-validation is performed usingsensitivity, specificity, balanced accuracy (BAC, i.e., average ofsensitivity and specificity), and area under the receiver operatingcharacteristic curve (AUC) as performance criteria. For sensitivity andspecificity, the mean and 95% confidence interval (CI) estimated from abinomial distribution are reported, and their average for BAC. For AUCboth the mean value and standard deviation (std) of the values obtainedare shown.

Image Sampling Strategies

The sampling operator selects N pixels inside a lesion and then itcenters a p×p pixel patch at each pixel location. For DIP detection,four sampling strategies are investigated. The first two arespecifically designed for blobs and curvilinear components,respectively, which are the typical structures seen inside a lesion(11). The other two, however, are not targeting any particular lesionstructure; yet, they result in excellent image classificationperformance.

Detector for blobs and corners: Blobs, dots, and globular structures arefrequently observed in a lesion. The scale-invariant feature transform(SIFT) (12) is used to detect these structures, a procedure also used in(11) (FIGS. 2A-2B).

Detector for curvilinear structures: the SIFT operator is not stable forridge detection (12) and it may fail to localize curvilinear structuresin the lesion, as it was noted also by Zhou et al (11). Instead, forcurvilinear structures, a Frangi filter (Frangi) is applied at threescales CT=1, 2, and 3 (10). Points with higher filter responses havehigher probabilities of being curvilinear structures. A Frangi filter issimilar to the Steger filter used in Zhou et al. (FIGS. 3A-3B)

Grid sampling: Sampling on a regular grid of size of g (Grid-g) placedon a lesion. When g is small, this is also called dense sampling.

Radial sampling: Sampling using polar coordinates on axes placed on thelesion with origin at the center of the lesion (Radial). The rationalebehind this scheme is that a lesion generally follows a radially growingpattern (14).

Feature Pooling Schemes

The popular average pooling and spatial pooling schemes areinvestigated. Average pooling uses averaging of the class assignmentsacross all patches. This is equivalent to building a normalizedhistogram, whereby each bin corresponds to a codeword in a codebook andthe bin's value is proportional to the number of patches assigned tothat codeword. Spatial pooling detects homogeneous regions inside alesion and then uses average pooling in each homogeneous region. Alesion is segmented into 3 to 8 regions using the normalized cut method(FIG. 4). Tiny regions are grouped with nearby larger ones. Thus, afterspatial pooling, a single vector (histogram) is produced for eachsegmented region. In the proposed method, a whole lesion is representedas a fully connected weighted graph, whose nodes correspond tohomogeneous regions. The weight of an edge is the Euclidean distancebetween the vectors of the two connected nodes (regions). Then a lesionis represented using six features implemented in the graph measuretoolbox (15), namely clustering coefficient, maximized modularity,characteristic path length, eccentricity for each vertex, radius, anddiameter of the graph (graph eccentricity, radius, and diameter are notthe same lesion measures defined in (2)). Tree and graph schemes havebeen proposed before (16-17), however, not for malignant classification.This proposed weighted graph model extends recent work in which anon-weighted graph lesion representation was employed for melanomadetection (18).

Codebook and Classifier Implementation Details

Codebooks are built using K-mean clustering on a set of patches obtainedby randomly sampling 1,000 patches from every lesion so that everylesion contributes equally to the codebook construction. Thus, theevaluation uses transductive inference (19), i.e., in this classifierlearning method labeled training data and unlabeled testing data wereemployed, while for testing labels were predicted for the latter. Thenumber of cluster is 200 for wavelet features and 100 for colorfeatures. The overall performance is not sensitive to these choices.Separate codebooks are built for wavelet and color, and differentcodebooks for the three patch sizes: 16, 24, and 32. By default, averagepooling is used, if not specified otherwise. This classifier usessupport vector machines (SVMs) with a χ² kernel, which is thestate-of-the-art setting for BoF model. For graph theory features, aGaussian kernel which is the common choice for SVMs is used. Thethreshold for the classifier's output was chosen by maximizing theaverage of sensitivity and specificity on the labeled training data. Forclassifier combination, simple ensemble averaging is used (weightedcombination (20) yielded very similar results on this dataset).

The Effect of Number of Patches Sampled

Choosing the same number of patches for all lesions is not reasonable,since lesions differ in size. Instead, a number of patches proportionalto that lesion's area were chosen. Simple grid sampling was used andgrid size was chosen from the set {1, 5, 10, 20, 40, 100}. Using a gridsize g is equivalent to sampling approximately (100/g²) % points from alesion. A square patch of 24 pixels in size is used. FIGS. 5A-5B showthat this percentage value affects significantly both performancemeasures. BAC starts to converge when the number of points approaches 4%of the lesion's area, while AUC converges earlier at about 1%. Thus,only 4% of points, i.e., Grid-5, from a lesion need to be sampledwithout decreasing performance significantly for both BAC and AUC.

The Effect of Sampling Strategy

Now four lesion sampling methods, Grid-5, Radial, SIFT, and Frangi, areconsidered and the parameters and thresholds of the latter three methodsare adjusted to retain 4% of all possible samples. In addition, theclassifiers from Radial, SIFT, and Frangi are combined with Grid-1(denoted as Com) to test whether classification accuracy improves whencombining classifier training with interest points located atdermoscopic structures instead of simply using all possible pointsalone, i.e., Grid-1. FIGS. 6A-6B show that regular grid sampling Grid-5provides results comparable to Radial, SIFT, and Frangi. A comparisonbetween Com and Grid-1 reveals only a marginal improvement in BAC, butno improvement in AUC, when incorporating the more complicated interestpoint detectors instead of using simple dense sampling alone.

The Effect of Sampling at Multiple Scales

For each sampling strategy, square patches of size 16, 24, and 32 pixelsare extracted, and the classifiers obtained from these three scales aregrouped. For the multi-scale model of Com, 12 classifiers are ensembledfrom four sampling methods and three scales. FIGS. 7A-7B shows thatmulti-scale sampling can improve the performance of some methodscompared to sampling at a single scale with patches of size 24. However,none of the multi-scale models in FIGS. 8A-8B is significantly betterthan Grid-1 using a single scale sampling.

The Effect of Spatial Pooling

Spatial pooling for patches of size 16×16 centered on every pixel areused, since it was observed empirically that a patch of size 16 performsbetter than size 24 or size 32 for graph theory features. Theclassifiers built from spatial pooling and Grid-1 are ensembled, and thecombined model is denoted as DenSpa. DenSpa is compared with Grid-1,Com, and the multiscale models of Grid-1 and Com, denoted as GridMuI andComMuI, respectively, in Table 3. DenSpa performs the best among thefive schemes in all measures. The mean sensitivity of the other fourmethods without spatial pooling is below the 95% CI of DenSpa. Theimprovement for specificity is not so significant, but the AUC of DenSpais significantly different from the other four methods as revealed by anunpaired t-test at 95% confidence level.

TABLE 3 Classification Performance Methods Sensitivity (95% CI) Fuzzyc-Means AUC (std) Grid-1 82.49 [79.16, 85.40] 83.32 [81.39, 85.07] 90.93± 2.67 GridMu1 82.31 [78.92, 85.18] 84.15 [82.28, 85.88] 91.16 ± 2.55Com 81.96 [78.67, 84.97] 84.68 [82.82, 86.37] 90.87 ± 2.60 ComMul 82.40[79.16, 85.40] 84.19 [82.37, 85.97] 90.99 ± 2.53 DenSpa 86.17 [83.11,88.80] 84.68 [82.82, 86.37] 92.71 ± 2.25

Example 4

Portable Library for Melanoma Detection: Comparison of SmartphoneImplementation with Desktop Application

The automated procedure for lesion classification is based on thebag-of-features framework (3,9) and comprises the main steps of lesionsegmentation, feature extraction, and classification.

Dataset

A total of 1300 artifact free images are selected: 388 were classifiedby histological examination as melanoma and the remaining 912 wereclassified as benign.

Image Segmentation

In addition to the lesion, images typically include relatively largeareas of healthy skin, so it is important to segment the image andextract the lesion to be considered for subsequent analysis. To reducenoise and suppress physical characteristics, such as hair in and aroundthe lesion that affect segmentation adversely, a fast two-dimensionalmedian filtering (21) is applied to the grey scale image. The image isthen segmented using three different segmentation algorithms, namelyISODATA (iterative self-organizing data analysis technique algorithm)(22), fuzzy c-means (23-24), and active contour without edges (25). Theresulting binary image is further processed using morphologicaloperations, such as opening, closing, and connected component labeling(26). When more than one contiguous region is found, additionalprocessing removes all regions except for the largest one. The endresult is a binary mask that is used to separate the lesion from thebackground.

Feature Extraction

Among the criteria employed by dermatologists to detect melanoma, asdescribed by Menzies rules (27) and the 7-point list (28), textureanalysis is of primary importance, since, among other things, malignantlesions exhibit substantially different texture patterns from benignlesions. Elbaum et al. (29) used wavelet coefficients as texturedescriptors in their skin cancer screening system MelaFindR and otherprevious work (9) has demonstrated the effectiveness of waveletcoefficients for melanoma detection. Therefore, this library includes alarge module dedicated to texture analysis.

Feature extraction works as follows: for a given image, the binary maskcreated during segmentation is used to restrict feature extraction tothe lesion area only. After placing an orthogonal grid on the lesion,patches of size K×K pixels were sampled repeated from the lesion, whereK is user defined. Large values of K lead to longer algorithm executiontime, while very small values result in noisy features. Each extractedpatch is decomposed using a 3-level Haar wavelet transform (30) to get10 sub-band images. Texture features are extracted by computingstatistical measures, like mean and standard deviation, on each sub-bandimage, which are then are put together to form a vector that describeseach patch.

Image Classification

A support vector machine (SVM) is trained using a subset (training set)of the total images available, and the resulting classifier is used todetermine whether the rest of the images, i.e., test set, are malignantor benign (3).

Training:

1. For each image in the training set,

(a) Segment the input image to extract the lesion.

(b) Select a set of points on the lesion using a rectangular grid ofsize M pixels.

(c) Select patches of size K×K pixels centered on the selected points.

(d) Apply a 3-level Haar wavelet transform on the patches.

(e) For each sub-band image compute statistical measures, namely meanand standard deviation, to form a feature vector F_(i)={m₁, sd₁, m₂,sd₂, . . . }.

2. For all feature vectors F_(i) extracted, normalize each dimension tozero mean and unit standard deviation.

3. Apply the K-means clustering (31) to all feature vectors F_(i) fromall training images to obtain L clusters with centers C={C₁, C₂, . . . ,C_(L)}.

4. For each training image build a L-bin histogram. For feature vectorF_(i), increment the jth bin of the histogram such that min_(j)C_(i)-F_(i).

5. Use the histograms obtained from all the training images as the inputto a SVM classifier to obtain a maximum margin hyperplane that separatesthe histograms of benign and malignant lesions.

The value of parameter M is a trade-off between accuracy and computationspeed. Small values of M lead to more accurate classification results,but computation time increases accordingly. When the algorithm runs onthe smartphone device, to reduce computation time, M=10 was chosen forgrid size, K=24 for patch size, and L=200 as the number of clusters inthe feature space. By exhaustive parameter exploration (9), it wasdetermined that these parameters are reasonable settings for thedataset. FIG. 8 summarizes in graphical form the feature extraction andclassification steps of the proposed procedure.

Testing: Each test image is classified using the following steps:

1. Read the test image and perform Steps 1(a)-(e) and 2 in the trainingalgorithm, to obtain a feature vector F_(i) that describes the lesion.

2. Build an L-bin histogram for the test image. For all feature vectorsF_(i) extracted from the image, increment the jth bin of the histogram,such that minj C_(i)-F_(i), where this time the cluster centers C_(i)are the centers identified in Step 3 of the training procedure.

3. Submit the resulting histogram to the trained SVM classifier toclassify the lesion.

For test images, likelihood of malignancy can be computed using thedistance from the SVM hyperplane. Training of the SVM classifier wasperformed off-line on a desktop computer, while testing is performedentirely on the smartphone device.

iPhone 4® Implementation

A menu based application is developed that implements the automatedprocedure outlined in the previous sections (FIGS. 9A-9D). The user cantake a picture of a lesion or load an existing image from the phonephoto library. The image is then analyzed on the phone in quasi realtime and the results of classification are displayed on the screen.

Comparison of the Segmentation Methods

To assess the performance of the proposed application the automaticsegmentation is compared with manual segmentation. For each image anerror, defined (32) as the ratio of the nonoverlapping area betweenautomatic and manual segmentation divided by the sum of the automaticand manually segmented images, was calculated. The error ratio is zerowhen the results from automatic and manual segmentation match exactly,and 100 percent when the two segmentations do not overlap. Thus theerror is always between zero and 100 percent, regardless of the size ofthe lesion. An earlier study found that when the same set of images wasmanually segmented by more than one expert, the average variability wasabout 8.5 percent (32). This same figure was used and included anadditional tolerance of 10 percent error to account for the large numberof images in the dataset. Therefore, the cutoff for error ratio was setto 18.5 percent, considering that a lesion is correctly segmented by theautomated procedure if the error ratio is less than 18.5 percent.

The dataset of 1300 skin lesion images was segmented using the threesegmentation techniques mentioned previously. Table 4 shows the numberof images correctly segmented, and the mean and standard deviation oferror for all images. The active contour method was found to be the mostaccurate, as it had the highest number of images segmented correctly andleast mean error ratio. ISODATA and Fuzzy c-Means, in that order,followed the active contour method in accuracy. FIGS. 10A-10C show theerror distribution of three segmentation methods, where the number ofimages is plotted against the error ratio. The threshold of 18.5% ismarked as the vertical dotted line. Of the 1300 images examined, 754images had error ratio below 8.5% (variability of error among domainexperts), 1147 images had error ratio below 18.5% (threshold for correctsegmentation), and 153 images with error ratio above 18.5%. Thisseemingly high error is due to the fact that manual lesion segmentationyields a smooth boundary, while automatic segmentation detects fineedges on the border.

TABLE 4 Performance of Segmentation Techniques Fuzzy Active ISODATAc-Means Contour Images correctly segmented 883 777 1147 Imagesincorrectly segmented 417 523 153 Mean Error 19.46% 20.40% 9.69%Standard Deviation Error 22.41 19.80 6.99

Classification Accuracy

10 trials of 10-fold cross validation were performed on the set of 1300images. The dataset was divided into 10 folds, nine folds with 39melanoma and 92 benign lesions and the remaining fold with 37 melanomaand 87 benign lesions. Of the 10 folds, nine were used for training andone was used for testing. 10 rounds of validation were performed whereeach fold was chosen for testing, to get 10×10=100 experiments. Anaverage over these 100 experiments demonstrated 80.76% sensitivity and85.57% specificity.

FIG. 11 shows the receiver operating characteristic (ROC) curve ofclassification computed from testing data (33), whereby the area underthe curve is 91.1%. The threshold to maximize the mean of sensitivityand specificity on the training set was chosen. The 95% confidenceinterval on testing data was estimated using a binomial distribution forsensitivity to be [77.1%, 83.9%] and for specificity was estimated to be[83.5%, 87.4%]. The classification accuracy is same on the desktopcomputer and iPhone 4® smartphone.

Execution Time

The time taken for active contour segmentation and classification on theApple® iPhone 4® smartphone is compared with a typical desktop personalcomputer (2.26 GHz Intel® Core™ 2 Duo with 2 GB RAM). The classificationtime includes time taken for feature extraction. The average image sizein the dataset is 552×825 pixels. Table 5 shows computation time inseconds for both platforms. For the largest image in the dataset whichhas dimensions 1879×1261 pixels, segmentation takes 9.71 sec andclassification takes 2.57 sec on the iPhone®. Thus, the whole analysisprocedure takes under 15 sec to complete. This proves that the libraryis light enough to run on a smartphone which has limited computationpower.

TABLE 5 Computation Time Mean time (sec) Apple ® IPhone 4 ® Desktopcomputer Segmentation 883 777 Classification 417 523

Example 5 Detection of Blue-Whitish Veil in Melanoma Using ColorDescriptors Dataset

There were 1,009 ELM skin lesion images collected from a widelyavailable commercial database (7), with full annotations for the ABCDrule and 7-point checklist. In this dataset, 252 images are benign, 757images are melanoma. Presence of 163 blue-whitish veil skin lesions inthis dataset were labeled by expert dermatologists.

Local Classification

For each local neighborhood of pixels, color histograms were computed,i.e. distribution of pixel intensities in various color models (RGB,HSV, YUV, O1O2O3, Nrgb). These color models are used because of theirinvariance to change in lighting conditions (34). These local featuresare used for detection of color in that local neighborhood.

Training: For all images belonging to the training set perform thefollowing steps:

1) Segment the input image to extract the skin lesion (ROI).

2) Perform transformation of the input RGB image to different colorspaces (O₁O₂O₃, HSV, Nrgb, YUV).

3) From ROI select non-overlapping patches of size K×K pixels.

4) Extract low-level color features from these K×K patches. For eachchannel in all color spaces build a separate P bin histogram H.

-   -   a) For all pixels belonging to the extracted patch, increment        the j^(th) bin of histogram where j=I_(c)/P×M_(c). I_(c) is        pixel intensity and M_(c) is maximum intensity in the specified        color space.    -   b) Concatenate all the extracted histograms to form a feature        vector F_(i)={H₁,H₂, . . . }, for a given patch in the image.    -   c) Based on prior knowledge of the patch color, mark the feature        vector as blue-whitish veil or not blue-whitish veil.

5) Perform step 4 for all the patches in the ROI to obtain F_(i)'s.

6) Input all F_(i) extracted from step 5 to linear SVM to obtain maximummargin hyperplane.

Testing: For all images belonging to the testing set perform thefollowing steps:

1) Perform steps 1-4 from training.

2) For each feature vector F_(i) belonging to a patch.

in the image use SVM to classify as blue-whitish veil or notblue-whitish veil.

3) Classify all extracted patches in the ROI.

Global-Level Classification: Approach 1

In the second step local classification features are used to perform aglobal-level classification. Experiments were performed with two choicesof global level classifiers. The first global classifier builds aprobability distribution of the local classification result. The secondglobal classifier uses a trivial approach to mark positive presence ofcolor, when one or more local neighborhoods have been marked withpresence of color.

Training: After applying patch level classification on all images in thetraining set, the following steps were performed:

1) For each patch in the training image perform patch-levelclassification to obtain probability estimate of the membership of thepatch to blue-whitish veil/not blue-whitish veil.

2) Build a B bin, global-level histogram G_(i) for each training image.For all patches in the training image:

a) Increment the j^(th) bin of the histogram G_(i), wherej=round(P_(es)/B), P_(es) is the probability estimate of a patchmembership to blue-whitish veil/not blue-whitish veil.

3) Perform steps 1 and 2 for all images in the training set to obtainhistogram G_(i).

4) Input all G_(i) obtained from step 3 to linear SVM to obtain maximummargin hyperplane.

Testing:

1) Perform steps 1 and 2 for a test image to obtain histogram G_(i).

2) Use SVM from the training to mark presence of blue-whitish veil inthe given test image.

Global-Level Classification: Approach 2

In this classifier a trivial approach is used, i.e., an image with oneor more blue-whitish patch is marked for positive presence ofblue-whitish veil. If none of the patches in the image have blue-whitishveil, then blue-whitish veil is absent from the image.

Classification Results

A set of 326 non blue-whitish veil lesions were selected randomly andwere combined with 163 blue-whitish veil lesion to form a subset. Thereason for subset selection is to have a proportionate representation ofthe both classes. For each trial of cross-validation a new subset of nonblue-whitish veil images was selected randomly from the whole dataset.

Ten trials of 10-fold cross validation were performed on a set of 489images. The dataset was divided into 10 folds, nine folds with 16blue-whitish veil lesions and 32 lesions where it is absent and theremaining fold with 19 blue-whitish veil lesions and 38 non blue-whitishveil lesions. Out of the 10 folds, nine are used for training and onefor testing. 10 rounds of validation were performed, such that each foldwas chosen for testing at least once. Therefore there were 10×10=100experiments. An averaged over 100 experiments was obtained whichdemonstrated sensitivity and specificity of the algorithm. FIG. 12depicts the results of blue-whitish veil detection on th skin lesionimages. Table 6 shows the classification accuracy of both global levelapproaches. It was observed that the trivial approach performs better.

TABLE 6 Classification Accuracy of blue-whitish veil detectionSensitivity Specificity Global-level Approach 1 90.59% 65.50%Global-level Approach 2 95.27% 63.90%

The low specificity of the blue-whitish veil detection is because oflarge false positives due to regression structures. Regressionstructures are one of the minor criterion of the 7-point checklist. Itis defined as associated white and blue areas which are virtuallyindistinguishable from blue-whitish veil (7). Experiments also wereperformed for detection of both blue-whitish veil and regressionstructures. Table 7 shows the classification accuracy of both globallevel approaches. It was observed that the specificity has increasedsubstantially because of lower false positives.

TABLE 7 Classification Accuracy of blue-whitish veil and regressionstructures Sensitivity Specificity Global-level Approach 1 95.64% 72.30%Global-level Approach 2 96.66% 68.28%

Parameter exploration was performed to find the most suitable choice ofthe non-overlapping square patch size used for extraction of local colorfeatures. FIG. 13A shows classification accuracy by varying the patchsize. It was observed that small patch size introduce noise and forlarge patch sizes the performance degrades because good discriminativelocal features can no longer be detected. It also was observed thatfirst global approach is more stable to the choice of patch size.

Parameter exploration was performed to find the most suitable choice forhistogram quantization of color models to represent local features. FIG.13B shows the classification accuracy with varying bin size oflocallevel color histograms. The first global approach depends upon thequantization of the global-level feature histogram. FIG. 14 illustratesthat smaller bin size of histogram has better specificity.

Simulation of Variance in Lighting Condition

Variance in lighting conditions was simulated by scaling and shiftingthe pixels intensity values in the dataset to show stability of thealgorithm. The pixels intensity was multiplied by a scaling factor thatvaries from 0.25 to 2.0. FIG. 15A shows that the classification accuracyis invariant to light intensity scaling. Illumination change also wassimulated by shifting pixels intensity values in the dataset images. Thepixels intensity was shifted by adding and subtracting a value thatvaries from −50 to 50. FIG. 15B shows that the classification accuracyis invariant to light intensity shifting.

Example 6 Instantiation of 7-Point Checklist on Smart Handheld DevicesDataset

Only images considered as low difficulty by the experts (7) were chosen.There were 385 low difficulty images in the database (7) and thesegmentation methods described herein could provide a satisfactoryboundary for 347 (90.13%) of them. In the selected set of 347 images:110 were classified by the 7-point list as melanoma and the remaining237 were classified as benign.

Feature Extraction

To identify a region of interest (ROI), an image is first converted togreyscale, and then fast median filtering (21) for noise removal isperformed, and followed by ISODATA segmentation (23), and severalmorphological operations. From the ROI, color and texture featuresrelating to each criterion on the 7-point checklist were extracted, asfollows.

I. Texture Features: They provide information on the various structuralpatterns (7) of 7-point checklist, such as pigmentation networks,vascular structures, and dots and globules present in a skin lesion.Haar wavelet coefficients (9) and local binary patterns (34) can beutilized for melanoma detection.

Haar Wavelet: From the ROI non-overlapping K×K blocks of pixels wereselected, where K is a user defined variable. Computation time forfeature extraction is directly proportional to the block size K. Theblock of pixels is decomposed using a three-level Haar wavelet transform(30) to get 10 sub-band images. Texture features were extracted bycomputing statistical measures, like the mean and standard deviation, oneach sub-band image, which are then combined to form a vector W_(i)={m₁,sd₁, m₂, sd₂, . . . }. Haar wavelet extraction for texture feature is asfollows:

1) convert the color image to greyscale; select a set of points in theROI using a rectangular grid of size M pixels.

2) Select patches of size KxK pixels centered on the selected points.

3) Apply a 3-level Haar wavelet transform on the patches.

4) For each sub-band image compute statistical measures, namely mean andstandard deviation, to form a feature vector W_(i)={m₁, sd₁, m₂, sd₂, .. . }.

5) For all feature vectors W_(i) extracted, normalize each dimension tozero-mean and unit-variance.

6) Apply K-means clustering (31) to all feature vectors W_(i) from alltraining images to obtain L clusters with centers C={C₁, C₂, . . . ,CL}.

7) For each image build an L-bin histogram H_(i). For feature vectorW_(i), increment the jth bin of the histogram such thatmin_(j)∥C_(j)−W_(i)∥.

The value of parameter M is a trade-off between accuracy and computationspeed. When the algorithm runs on a handheld device to reducecomputation time, M=10 for the grid size, K=24 for patch size, and L=200as the number of clusters in the feature space were chosen. Aspreviously demonstrated, these parameters are reasonable settings (9).

Local Binary Pattern (LBP): LBP is a robust texture operator (35)defined on a greyscale input image. It is invariant to monotonictransformation of intensity and invariant to rotation. It is derivedusing a circularly symmetric neighbor set of P members on a circle ofradius R denoted by LBP_(PR) ^(riu) (35). The parameter P represents thequantization of angular space in the circular neighborhood, and Rrepresents the spatial resolution. A limited number of transitions ordiscontinuities (0/1 changes in LBP) are allowed to reduce the noise andfor better discrimination of features. The number of transitions in LBPwere restricted to P, and transitions greater than that are consideredequal. An occurrence histogram of LBP with useful statistical andstructural information is computed as follows:

1) Convert the color image to greyscale.

2) Select pixels belonging to ROI and compute local binary patternLBP_(PR) ^(riu) (35).

3) Build an occurrence histogram, where the j^(th) bin of the histogramis incremented, if the number of transitions in LBP is j.

4) Repeat steps 2 and 3 for all pixels in ROI.

The occurrence histograms for LBP_(16,2) and LBP_(24,3) were built andconcatenate them to form a feature vector L_(i).II. Color Features: Detection of the 7-point checklist criteria, such asblue-whitish veil and regression, which consist of mixtures of certaincolors, can be achieved by analyzing the color intensity of pixels inthe lesion (36). To reduce the variance due to the lighting conditionsin which dermoscopic images were taken, the HSV and LAB color spaceswere considered also, which are invariant to illumination changes (34).

Color Histograms: To extract the color information of a lesion, a colorhistogram was computed from the intensity values of pixels belonging tothe ROI. Additional images in the HSV and LAB color spaces are obtainedfrom the original RGB image. The intensity range of each channel isdivided into P fixed-length intervals. For each channel a histogram wasbuilt to keep count of the number of pixels belonging to each interval,resulting in a total of nine histograms from three color spaces.Statistical features, such as standard deviation and entropy (Eq. 9), ofthe nine histograms are also extracted as features for classification.More specifically, entropy is defined as

$\begin{matrix}{{entropy} = {\sum\limits_{i = 1}^{P}\; {f\left( {{histogram}\lbrack i\rbrack} \right)}}} & \left( {{Eq}.\mspace{14mu} 9} \right)\end{matrix}$

where histogram[i] is the normalized pixel count of i^(th) bin and

${f(n)} = \left\{ \begin{matrix}{n \times \log \; 2(n)} & {{{if}\mspace{14mu} n} > 0} \\0 & {{{if}\mspace{14mu} n} = 0.}\end{matrix} \right.$

The color histogram feature extraction steps are as follows:

1) Obtain skin lesion image in HSV and LAB color space from input RGBimage

2) For each channel in all three color spaces build a separate P binhistogram

3) For all pixels belonging to ROI, increment the j^(th) bin ofhistogram where j=I_(c)/P×M_(c). I_(c) and M_(c) are pixel intensity andmaximum intensity in the specified color space.

4) Compute the standard deviation and entropy of the histogram.

5) Repeat steps 3 and 4 for all the channels in RGB, HSV, and LAB colorspace. Color histogram and statistical features are combined to form afeature vector C_(i).

Classification

The features from color histogram C_(i), Haar wavelet H_(i), and LBPL_(i) are combined to form F_(i)={C_(i), H_(i), L_(i)}. For eachcriterion in the 7-point checklist a filtered feature selection wasperformed to obtain a subset of F_(i) with the highest classificationaccuracy. Correlation coefficient values (between F_(i) and eachcriterion) are used as the ranking criterion in the filters. The size ofthe subset and the parameters of the linear support vector machine (SVM)are obtained by grid search. Each criterion requires both training andtesting.

Training: The training algorithm is as follows:

1) Segment the input image to obtain region of interest.

2) Extract Color histogram, Haar wavelet, and Local binary pattern.Concatenate them to form feature vector F_(i).

3) Repeat steps 1 and 2 for all training images.

4) Perform filter feature selection to choose a subset of features S_(i)from F_(i).

5) Input S_(i) to linear SVM to obtain maximum margin hyperplane.

Testing: Image classification is performed as follows:

1) Read the input image and perform steps 1 and 2 from the trainingalgorithm.

2) For each criterion use the SVM coefficients obtained from training tomake a prediction.

3) Scores from all major and minor criteria are summed up. If the scoreis greater than or equal to 3 then lesion is classified as melanoma,otherwise as benign.

Classification Results

Generally, the end user can take an image of the skin lesion using the 5megapixel built in camera with LED flash, or load the image from medialibrary. The image is analyzed in quasi real time and the final resultdisplayed on the screen. FIGS. 16A-16B depict the menu based applicationin use on an Apple® iPhone® device showing an image of a skin lesionacquired by the device with the options of choose image, take photo or7-Point Rule. The next screen displays the scores for each of the 7criteria, a total score and a diagnosis based on the same.

A 10-fold cross validation was performed on the set of 347 images totest the menu based application by comparing the classification accuracyof each criterion separately against the overall final classification byexpert physicians (7). The dataset was divided into 10 folds, nine foldswith 11 melanoma and 23 benign lesions and the remaining fold with 11melanoma and 30 benign lesions. Of the 10 folds, nine were used fortraining and one was used for testing. 10 rounds of validation wereperformed where each fold was chosen for testing and the rest were fortraining, to get 10 experiments. The classification accuracy of eachcriterion was compared and the overall decision of the 7-point checklistwith dermatology and histology.

Table 8 presents the sensitivity and specificity of the algorithm inclassification of each of the 7-point checklist criterion. There waslower accuracy for the regression structures, because they are usuallyindistinguishable from the blue-whitish veil via dermoscopy (7).However, this is not an issue, as is it only necessary to obtain aminimum score of 3 to correctly detect a melanoma.

TABLE 8 Classification for all Criteria Sensitivity Specificity AtypicalPigment Network 72.86% 70.40% Blue-Whitish Veil 79.49% 79.18% AypicalVascular Pattern 75.00% 69.66% Irregular Streaks 76.74% 79.31% IrregularPigmentation 69.47% 74.21% Irregular Dots and Globules 74.05% 74.54%Regressive Structures 64.18% 67.86%

In Table 9 the sensitivity and specificity of the algorithms is comparedwith the decision made by expert clinicians via dermoscopy. Table 10presents the confusion matrix computed using the sum of ten confusionmatrices from the ten test sets, given from the 10-fold crossvalidation. Another classification experiment using SVM also wasperformed, where the 7-point checklist was ignored and each skin lesionwas directly classified as melanoma or benign. The feature vectors,feature selection scheme, and final ground truth (melanoma/benign) arethe same as the classification using the automated 7-point checklist.Table 5 shows that classification accuracy is much lower when the7-point checklist is ignored.

TABLE 9 Classification Accuracy Sensitivity Specificity 7-PointChecklist 87.27% 71.31% Ignoring 7-Point Checklist 74.78% 70.69%

TABLE 10 Confusion Matrix of the Automated Decision Predicted ConfusionMatrix Melanoma Benign Dermoscopy Melanoma 96 14 Benign 68 169

Execution Time

The time needed for classification using the ISODATA segmentationalgorithm (23) on the Apple® iPhone 3G® is compared with a typicaldesktop computer (2.26 GHz Intel® Core™ 2 Duo with 2 GB RAM, IntelCorporation). The average image size in the dataset is 552×825 pixels.The classification time includes time taken for feature extraction.Table 11 shows computation time in seconds for both platforms. It can beseen that the whole procedure takes under 10 sec to complete therebydemonstrating that the application is light enough to run on asmartphone which has limited computation power.

TABLE 11 Mean Computation Time Mean time (sec) Apple ® IPhone 3G ®Desktop Computer Segmentation 2.3910 0.1028 Classification 7.4710 0.2415

Example 7 Implementation for Buruli Ulcers

FIGS. 17A-17B are flowcharts depicting the algorithm steps in thesegmentation, feature extraction and classification modules foridentification/classification of an object of interest, for example, aBuruli ulcer, as in the below example, or a melanoma.

Segmentation

Both color and luminance are important characteristics for Buruli lesionsegmentation. The key idea of the segmentation method simply starts byconsidering the common foreground and background obtained by theluminance and color components as lesion and skin, respectively, andthen applying a supervised classifier to the remaining pixels is key.Segmentation 100 comprises the following steps.

First, contour initialization 110 comprises the following steps:

1) Read input RGB image at 111;

2) Transform RGB image to four other color spaces: La*b*, HSV, YCbCr,Lu*v* at 112;

3) Apply Otsu's thresholding method (1) to eight color channels: a*, b*,H, S, Cb, Cr, u*, v*, to obtain eight segmentation masks at 113.

4) Fuse these eight masks by a voting system to form a new mask at 114.For each pixel, if more than three masks agree to be foreground, then itis classified as a lesion pixel;

5) Draw a convex hull at 115 which covers the fused mask to be theinitialized contour for the following steps.

Secondly, contour evolution 120 comprises the following steps:

1) For each segmentation mask obtained from the eight color channels,calculate the correlation coefficient with the fused mask at 121;

2) Apply the Chan-Vese Level Set segmentation method (2) for the colorchannel which has the largest correlation coefficient, to obtain a maskbased on color information M_(c) at 122.

Basically, in Chan-Vese, given an image I⊂Ω, the region-based activecontour model (37) assumes that image I is formed by two regions ofapproximately piecewise constant intensity c1 and c2 separated by acurve C, which minimizes the energy-based objective function:

$\begin{matrix}{{{\text{?}\left( {c_{1},c_{2},C} \right)} = {{{\mu \cdot {length}}\mspace{14mu} (C)} + {\lambda_{1}\text{?}\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {{{{I_{i}(x)} - c_{1,i}}}^{2}{x}}}} + {\lambda_{2}\text{?}\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {{{{I_{i}(x)} - c_{2,i}}}^{2}{x}}}}}}{\text{?}\text{indicates text missing or illegible when filed}}} & \left( {{Eq}.\mspace{14mu} 10} \right)\end{matrix}$

where the parameters μ>0 and λ₁,λ₂>0 are positive weights for theregularizing term and the fitting term, respectively. When applying thelevel set approach (38), the curve C can be represented as the zerolevel set C(t)={(x)|φ(t, x)=0} of a higher dimensional level setfunction Φ(t, x). Then the energy function can be rewritten as

$\begin{matrix}{{{E\left( {\Phi,c_{1},c_{2}} \right)} = {{\mu \cdot {\int_{\Omega}{\text{?}(\Phi){{\nabla\Phi}}\ {x}}}} + {\int_{\Omega}{\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {\lambda_{1}{{{{I_{i}(x)} - \text{?}}}\ }^{2}{H(\Phi)}{x}}}}} + {\int_{\Omega}{\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {\lambda_{2}{{{{I_{i}(x)} - \text{?}}}\ }^{2}\left( {1 - {H(\Phi)}} \right){x}}}}}}},{\text{?}\text{indicates text missing or illegible when filed}}} & \left( {{Eq}.\mspace{14mu} 11} \right)\end{matrix}$

where H is the Heaviside function. The evolution of is governed by thefollowing motion partial differential equation (PDE):

$\begin{matrix}{{\frac{\partial\Phi}{\partial t} = {\text{?}{(\Phi)\left\lbrack {{\mu \text{?}} - {\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {\lambda_{1}{{{I_{i}(x)} - \text{?}}}^{2}}}} + {\frac{1}{N}{\sum\limits_{i = 1}^{N}\; {\lambda_{2}{{{I_{i}(x)} - \text{?}}}^{2}}}}} \right\rbrack}}}{\text{?}\text{indicates text missing or illegible when filed}}} & \left( {{Eq}.\mspace{14mu} 12} \right)\end{matrix}$

where δ(Φ) is a regularized version of the Dirac delta function. Theevolution can be solved using finite differences, by updating eachc_(1,i) and c_(2,i) by the average of channel I, calculated inside (C)and outside (C).

3) Transform RGB image to gray scale image at 124; and

4) Apply the Chan-Vese Level Set segmentation method for the gray scaleimage, to obtain a mask based on luminance M_(i) at 125.

Thirdly, pixel classification 130 comprises the following steps:

1) For each pixel at 131, if it belongs to the common foreground ofM_(c) and M_(i), then it is classified as a foreground pixel or if itbelongs to the common background of M_(c) and M_(i), then it isclassified as a background pixel or it remains to be determined;

2) From the common background and foreground of M_(c) and M_(i),randomly sample 5000 pixels respectively, train a linear SVM using RGBand Lu*v* value at 132;

3) For each remaining pixel at 133, its RGB and Lu*v* value are used asinput for the classifier to obtain a decision on this pixel beingbackground or foreground.

Feature Extraction and Classification

As the first step, image sampling is a critical component whenbag-of-features methods are used for image classification. Thealgorithms provided herein enable more patches from dermoscopic interestregions to be sampled based on saliency values. Given a patch, saliency(39) is defined as shown in Eqs. 5-7.

Feature extraction 200 comprises the following steps:

1) Read input for the RGB image and the segmentation result in theregion of interest (ROI) at 201;

2) Extract color moment and wavelet coefficients for each patch insideROI and assign a corresponding cluster number for each patch at 202;

3) Calculate saliency value according to Eqs. (5-7) for each patchinside ROI at 203;

4) Use k-means clustering to separate the lesion into two regions basedon saliency values at 204;

5) Calculate the average intensity for separate regions respectively at205. Region with higher intensity is denoted as R_(h), and R_(l) is forregion with lower intensity;

6) Decide, at 206, sampling percentage for two separate regions by

P _(i)=(α·A _(i) /α·A _(i) +A _(h))×100%  (Eq. 13),

where P_(h) is the percentage of patches sampled from R_(h), A_(h) andA_(l) are area of R_(h), and R_(l) respectively; α is a coefficient tocontrol the percentage, here α is set to be 1.5; and

7) Randomly sample patches from R_(h), and R_(l) by correspondingsampling percentages and extract the bag-of-feature representation foreach lesion at 207.

Classification 300 comprises the steps of:

1) Training for SVM using manually segmented images with known labelsfor Buruli and non-Buruli ulcers at 301; and

2) Extracted features are used as input at 302 for the classifier toobtain a decision whether or not the lesion is a Buruli lesion.

Example 8 Obtention of Dermoscopic Images for Detection of Buruli Ulcers

Images were 24 bit full color with typical resolution of 4320×3240pixels. Data were collected in endemic BU communities of Cote d'Ivoireand Ghana with the help of local collaborators to the project thatincluded medical doctors, District Surveillance Officers, and communityhealth workers, using a DermLite II Multi-Spectral device(www.dermlite.com) for image acquisition. The device could provide whitelight for crosspolarization epiluminescence imaging, blue light forsurface coloration, yellow light for superficial vascularity, and redlight for deeper coloration and vascularity, using 32 bright LEDs, eightper color. This device was attached to a Sony Cybershot DSC-W300high-resolution camera, which provided a resolution of 13.5 MP. Thestudy has received IRB approval from the Human Subjects ProtectionCommittee at the University of Houston, as well as in Ghana and IvoryCoast, and all subjects and their parents gave written informed consentto the study in their native language.

Example 9 Application of Segmentation Scheme to Suspected Buruli Lesions

A set of dermoscopic images of 26 suspected BU lesions were obtained asdescribed herein. In the preprocessing step, images were firstdownsampled to 1080×810 pixels, and then processed with a 5×5 medianfilter and a Gaussian lowpass filter of the same size to removeextraneous artifacts and reduce the noise level. For postprocessing,morphological filtering was applied, and a distance transform (40) wasused to make the borders smoother. As ground truth for the evaluation ofthe border detection error, for each image, manual segmentation wasperformed by a field expert in Africa just after acquisition. Threedifferent metrics were used to quantify the boundary differences, namelyXOR error rate (XER) (41), true detection rate (TDR), and false positiverate (FPR) (11), defined as follows,

XER(A;M)=(A M)=M 100%

TDR(A;M)=(A\M)=M 100%

FPR(A;M)=(A\M)=M 100%; (Eq. 14)

where A denotes the area of automatic segmentation and M denotes themanual segmentation area obtained by the expert.

Images illustrating segmentation schemes are shown in FIGS. 18A-18D.Particularly, FIG. 18A shows the original lesion with the manualsegmentation from the expert. The lesion consists of two main parts:central areas with variegated distinctive colors, and the surroundingerythematous areas which exhibit a smooth transition to normal skins.Also, the complex texture of normal skin caused by the infected skinmakes the segmentation task more challenging. FIG. 18B shows the initialcontour obtained by the fusion of thresholding segmentations fromdifferent color channels. The initial mask covers the most significantlesion colors. FIGS. 18C-18D present the segmentation results aftercontour evolution in the color and luminance components, respectively.It is obvious that the segmentation in the color channel is good atdetecting the central area of a lesion with significant colors andmisses the surrounding areas, while segmentation in the luminancechannel is able to find the surrounding area, but always includes partof normal skin because of the smooth transition. The combination ofcolor and luminance information by pixel classification is shown in FIG.18E, while FIG. 18F presents the final segmentation result aftermorphological postprocessing. The latter is close to the expert'ssegmentation and detects both parts of the lesion successfully.

Comparison of Segmentation with Other Methods

The proposed segmentation method (based on Fusion and Classification,FC) was compared with three popular methods applied to skin lesionsegmentation, namely adaptive thresholding (AT) (7), gradient vectorflow (GVF) (8), and level set (LS) (11) segmentation. The initializationof contour for GVF and LS were both completed by the first step of thesegmentation scheme. For GVF snake, the elasticity, rigidity, viscosity,and regularization parameters were α=0:05, β=0:01, γ=1, and k=0:6,respectively. The maximum iteration number was 75. The LS method wasprocessed in the L*a*b* color space, using parameters λ₁=1, λ₂=1; andμ=0:1. The maximum number of iterations was 150. For this segmentationscheme, the same parameters as in the LS method were used for thecontour evolution step, where 5000 foreground and 5000 background pointswere randomly sampled to train the classifier. The segmentation resultsobtained are shown in FIGS. 19A-19D. Among these approaches, the AT andLS methods were disturbed by the illumination of the surrounding normalskins, the GVF method converged to some noisy or spurious edge points,while the method described herein successfully detected both the centraland surrounding areas of the lesion, resulting in an accurate border.

To quantify the performance of different segmentation methods, threedifferent metrics, namely XER (41), TDR, and FPR (42) were used tomeasure the segmentation accuracy, as described (43). XER is computed asthe number of pixels for which the automatic and manual borders disagreedivided by the number of pixels in the manual border. It takes intoaccount two types of errors: pixels classified as lesion by the expertthat were not classified as such by the automatic segmentation andpixels classified as lesion by the automatic segmentation that were notclassified as such by the expert, while the TDR method focuses on theformer and the FPR focuses on the latter, respectively. Table 12 showsthe segmentation performance of the different methods.

TABLE 12 Segmentation performance of different methods Methods XER (std)TDR (std) FPR (std) AT 39.46 ± 26.14 84.84 ± 17.22 24.30 ± 00   GVF24.01 ± 12.02 79.10 ± 12.97 4.17 ± 4.08 LS 26.54 ± 19.78 90.06 ± 8.44 16.60 ± 21.42 FC 19.25 ± 9.28  85.70 ± 9.86  5.15 ± 5.3 

The LS method can achieve the highest TDR at the cost of a higher FPR,because it always includes lesions and part of normal skin. On thecontrary, the GVF method performs the best in FPR at the cost of missingsome actual lesion areas. Overall, the segmentation method providedherein can achieve the best XER while keeping a relatively high TDR andlow FPR, and outperform other state-of-art segmentation methods inBuruli lesion images.

Example 10 A Classifier for Automatic Detection of Buruli Lesions

A set of dermoscopic images of 58 lesions, in which 16 lesions wereconfirmed BU and 42 lesions were non-BU lesions were obtained, asdescribed herein. Images were first downsampled to 1080 Å˜810 pixels,then manual segmentation of all lesions was applied to ensure thatevaluation of classification performance was not affected by possiblediscrepancies in the automated identification of the lesion boundaries.The default setup for bag-of-features was as follows: patches weresampled on a regular grid of 5 Å˜5, with patch size 24 Å˜24 pixels;color moments and wavelet coefficients were the patch descriptors; thecodebook was generated by k-means clustering with a size of 50codewords; an SVM classifier with RBF kernel was used for the finalclassification step. Leave-One-Out cross-validation was implemented toevaluate the performance of the method. Performance criteria includedsensitivity, specificity, and balanced accuracy (BAC, i.e., average ofsensitivity and specificity).

Codeword Representation

The idea of bag-of-features is that the large set of collected samplescan be automatically arranged into sub-clusters sharing similar colorand texture patterns. If some of these image patterns, i.e., clustercentroids, are distinctive, then the distributions of image patternswhich represent the skin lesions can have strong discriminative power.In other words, if the common image patterns of BU images aredistinguishable enough from those patterns of non-BU images, then thebag-of-features method can be a good way to classify BU and non-BUimages.

FIGS. 18A-18B show the shared image patterns of BU and non-BU lesions,respectively. The collection of samples from BU and non-BU images isclustered into 15 subclasses, and the patches that are closest to thecluster centroids are displayed. Most of the BU image patterns are lightin color and homogeneous in texture, corresponding to discolored andnecrotic skin, while non-BU patterns are darker and have more complextextures.

Effect of Sampling Strategies

One of the main parameters governing classification accuracy andprocessing time is the number of patches sampled. Since lesions differin size, the number of patches proportional to that lesion's area waschosen. Regular grid sampling and random sampling were applied with apatch size of 24×24 pixels, respectively. Grid sampling extractedpatches on a regular grid with size chosen from the set {1, 2, 5, 10,20, 50, 100}. Using a grid size g is equivalent to samplingapproximately (100/g²) % points from a lesion. Random sampling sampledpatches randomly with the corresponding percentage. FIG. 21A shows theclassification accuracy for different patch numbers. For both grid andrandom sampling, accuracy increases significantly as more patches aresampled, but it starts to converge when more than 4% of patches aresampled. Thus, only 4% of points need to be sampled from the lesion toachieve a maximum accuracy, but in substantially shorter time.

Patch size is another factor that can affect time and accuracy. Squarepatches of a size chosen from the set {8, 16, 24, 32, 40} on a grid ofsize 5×5 were extracted. FIG. 21B illustrates the impact of patch sizeon classification performance. A medium patch size of 24×24 pixelsachieved the best performance in our experiments. Small patches can beprocessed very fast, but they capture less information, while largepatches provide very good sensitivity. However, patches of very largesize ignore some details of local characteristics and result in muchhigher computational complexity.

Effect of Patch Descriptors

In the bag-of-features method, patch descriptors are used tocharacterize image patches and to discover similar patterns acrossimages. Different patch descriptors were tested, i.e., color moments andwavelet coefficients individually, as well as the combination of thesetwo. Color moments captured color and shape information, and waveletcoefficients captured texture-related features. Table 13 shows that bothsingle descriptors can achieve an accuracy around 80%, but that thecombination can make a significant improvement to 95%, indicating thatboth color and texture are important to discriminate BU from non-BUimages.

TABLE 13 Classification Performance of Different Patch DescriptorsDescriptors Sensitivity (%) Specificity (%) Accuracy (%) Color 87.5069.05 78.27 Texture 87.50 83.33 85.41 Combined 100 90.48 95.24

Effect of Codebook Size

The number of codebook centers is another factor that affects theperformance of bag-of-feature methods. Five codebook sizes were chosenfrom the set {10, 25, 50, 100, 200}. When the codebook size is small,patches are assembled into fewer groups, therefore the discriminativepower is not strong. As patches are grouped into more clusters, theaccuracy also increases; however, when the codebook size becomes toolarge, the dimension of the feature vector is also very large, so theoverall performance decreases because of over-fitting.

Effect of SVM Kernels

The performance of different types of SVM kernels was investigated.Table 14 shows that the performance of the linear kernel is the worst,while the nonlinear RBF and the chi square kernels, which map thefeature vector to a higher dimension feature space, can achieve betterperformance.

TABLE 14 Classification Performance of Different SVM Kernels KernelsSensitivity (%) Specificity (%) Accuracy (%) Linear 81.25 83.33 82.29RBF 100 90.48 95.24 Chi-square 100 88.10 94.05

Depiction of Buruli Ulcer on a Smart Device

The algorithms described herein can detect and diagnose a Buruli ulcerin early or late stage (FIG. 22A). FIG. 22B illustrates the grouping ofearly and late lesions obtained from the bag-of-features and featurehistograms created from wavelet and color moment features.

Example 11 Implementation for Multispectral Imaging

Lights of different frequencies can penetrate different skin depths. Forinstance, blue light with a shorter wavelength of about 470 nm formsimages of surface coloration, yellow light of about 580 nm forms imagesfor superficial vascularity and red light with a longer wavelength ofabout 660 nm penetrates deeper and visualizes deeper vascularity. Inthis example, the algorithms are applied to the classification of aBuruli ulcer. However, this algorithmic process is applicable to theidentification and classification of other objects of interest, asdescribed herein.

Architecture Overview

The architecture 400 (FIG. 23) for the extension and control of theprocessing chain for multispectral images comprises a framework 410having the primary tiers Script Manager 412 and Processing Engine 415 onan application programmer interface (API) platform 420 with associatedhardware 430. The Script Manager tier handles the configuration andexecution of the DSL scripts 414 that represent process chains. Aprocess chain encapsulates the particular execution steps required foranalysis of a category of skin lesion, as described herein. Processingchains, comprising one or more process stages, are described using a DSLdesigned for skin-lesion image processing. Each process stage mayconsist of one or more image processing modules (IMPs), which aretypically implemented in the C or C++ programming language. The DSLexposes these processes in a manner that allows an end user to chainIMPs, either in serial or parallel, without having intimate knowledge ofthe IMP implementation or the programming language that was used todevelop it.

The following is an example of a DSL implementation of the processchain:

Define_chain “classifier_rule_chain”, image do  artificat_removal hair_removal  segmentations = in_parallel do   fuzzy_c_means  active_contours  end  segmentation = score_and_return segmentations with segmentation do   extract_features   classify_lesion  end end

Process chains are completely configurable with only changes to the DSLscripts, allowing users to quickly try several analysis approaches. IMPscan be added to the system by developers who have minimal knowledge ofthe overall framework. Process chains can include other chains, so thatit is possible for example to run a skin cancer and a Buruli analysis onthe same lesion at the same time

The processing engine 415 comprises a script processor 416 and stagecomponents 418. The processing engine executes the preset scripts in thescript processor, returning the results to the script manager. Theprocessing engine is responsible for reading and interpreting thescript, managing the script as it runs and instantiating process steps,as required. Also, the processing engine interfaces with the underlyingoperating system's API 420 to facilitate the use of native processingcapabilities, including process and memory management and userinteraction.

Segmentation

Segmentation is performed as per process 100 described herein.

Feature Extraction

In feature extraction color moments for white light images, histogram ofintensities for multispectral images, and texture properties for bothfrom an object of interest, such as, but not limited to, a Buruli ulcerare extracted. Extracted features are used as input to support vectormachine (SVM), which outputs the classification, such as whether thelesion is Buruli or not or whether the lesion is malignant or not.

Feature extraction 500 (FIG. 24) comprises the steps of:

1) Read at 501 the input white light image (RGB) and segmentation result(region of interest (ROI));

2) Read at 502 the input multispectral images of blue, yellow, and redchannel, and transform to gray scale images;

3) Use white light image as a reference image, do image registration formultispectral images by maximizing mutual information at 503;

4) Extract bag-of-feature representation within ROI from the white lightimage with wavelet coefficients and color moment, respectively at 504;

5) Extract bag-of-feature representation within ROI from multispectralimages with wavelet coefficients and histograms, respectively, at 505;and

6) Pool features from the white light image and multispectral imagestogether, and perform feature selection to choose relevant features at506.

Classification

Classification (FIG. 24) is performed as per process 300 describedherein.

Example 11 Optical Skin Model

Skin tissues have different absorption and scattering properties whenlights of different frequencies passing through different skin layers.The attenuation coefficients of epidermis and dermis are related tothree parameters, e.g. the volume fractions of melanin, blood, andoxygenation respectively. Provided herein is an implementation of analgorithm for multispectral image classification based on the followingoptical skin models.

I _(det)(λ)*I _(calibration)(λ)⁼ S*A _(epi)(λ)² *A _(dermis)(λ)  (eq.15)

where λ is the wavelength, I_(det)(λ) is the detected intensity at eachpixel for each wavelength, I_(calibration)(λ) is the calibration factor,A_(epi)(λ) is the attenuation of the light intensity after passingthrough the epidermis, and A_(dermis)(λ) is the attenuation of lightintensity passing through dermis.

Here, A_(epi)(λ) is related to the volume fraction of melanin. It can bedetermined by,

λ_(epi)(λ)=exp[−μ_(a(epi))(λ)t┐  (eq. 16)

where t is the thickness of epidermis, which can be considered to beconstant as 0.6 mm, and

μ _(a(epi))(λ)=V_(mel)μ_(a(mel))(λ)+(1−V_(mel))μ_(a(skin))(λ):  (eq. 17)

where μ_(a(mel)) is the melanin absorption coefficient, and μa(skin) (λ)is the absorption coefficient of normal skin. These two coefficients areknown parameters. The remaining variable is the volume fraction ofmelanin in the epidermis V_(me).

In addition, A_(dermis(λ)) is related to volume fraction of blood in thetissue and the percent of that blood that is oxygenated. It can bedetermined by,

A _(dermis)(λ)=1.06−1.45[μ_(a(dermis))(λ)/μ′_(s)(λ)]^(0.35)  (eq. 18)

where μ′_(s)(λ) is the reduced scattering coefficient of the dermis,which is determined by the wavelength and

μ_(a(dermis))(λ)=V _(blood)μ_(a(blood))(λ)+(1−V_(blood))μ_(a(skin))(λ)  (eq. 19)

where V_(blood) is the volume fraction of blood in the dermis layer and,

μ_(a(blood))λ)=V _(oxy)μ_(a(oxy))(λ)+(1−V _(oxy))μ_(a(deoxy))(λ)  (eq.20)

where V_(oxy) is the fraction of blood that is oxygenated, μ_(a(oxy))(λ)and μ_(a(deoxy))(λ) are the absorption coefficients of HbO₂ and Hbrespectively. So the three remaining variables are: V_(mel), V_(blood),and V_(oxy). By inserting Eq. 16-20 into Eq. 15, and using intensitiesobtained from three different channels, three unknown physiologicalparameters V_(mel), V_(blood), and V_(oxy) can be solved.

Segmentation

Segmentation is performed as per process 100 described herein.

Feature Extraction

Feature extraction 700 (FIG. 25) comprises the steps of:

1) Read at 601 input white light image (RGB) and segmentation result(region of interest (ROI));

2) Read at 602 input multispectral images of blue, yellow, and redchannel, and transform to gray scale images;

3) Use white light image as a reference image, do image registration formultispectral images by maximizing mutual information at 603;

4) For each pixel within ROI, solve V_(mel), V_(blood), and V_(oxy) byEqs. 5-7 to reconstruct maps of melanin, blood, and oxygenatingpercentage at 604;

5) Extract bag-of-feature representation within ROI from thereconstructed maps with wavelet coefficients and histograms,respectively, at 605; and

6) Pool features from reconstructed images and perform feature selectionto choose relevant features at 606.

Classification

Classification (FIG. 25) is performed as per process 300 describedherein.

The following references are cited herein.

-   1. Jemal et al. CA: A cancer Journal for Clinicians, 60(5):277,    2010.-   2. Ganster et al. IEEE Transactions on Medical Imaging,    20(3):233-239, 2001.-   3. Csurka et al. Workshop on Statistical Learning in Computer    Vision, ECCV, 22(1), 2004.-   4. Morris and Guilak, Pervasive Computing, IEEE, 8(2):57-61,    April-June 2009.-   5. Logan et al. American Journal of Hypertension, 20(9):942-8, 2007.-   6. Hicks et al. Wireless Health 2010, WH'10:34-43, New York, N.Y.,    USA 2010.-   7. Argenziano et al. Dermoscopy: A Tutorial, Vol. 12, February 2000.-   8. Argenziano et al. J Am Acad Dermatol, 48(5):679-93, 2003.-   9. Situ et al. Annual International Conference of the IEEE    Engineering in Medicine and Biology Society, 1:3110, 2008.-   10. Yang et al. IEEE Conference on Computer Vision and Pattern    Recognition (CVPR), IEEE, 2009.-   11. Zhou et al. Biomedical Imaging: From Nano to Macro, 2009,    IEEE:1318-1321, 2009.-   12. Lowe, D. G., International Journal of Computer Vision,    60(2):91-110, 2004.-   13. Frangi et al. Medical Image Computing and Computer-Assisted    Interventation-MICCAI 98:130, 1998.-   14. Zhou et al. Biomedical Imaging: From Nano to Macro, 2008, 5^(th)    IEEE:1318-800-803, 2008.-   15. Rubinov, M. and Sporns, O., Neuroimage, 2009.-   16. Cucchiara, R. and Grana, C., Knowledge-Based Intelligent    Information Engineering Systems and Allied Technologies: Kes 2002:    166, 2002.-   17. Sadeghi et al. Proceedings of SPIE, 7623:762312, 2010.-   18. Situ et al. Annual International Conference of the IEEE    Engineering in Medicine and Biology Society, 2010.-   19. Vapnik, V., Statistical Learning Theory, Vol. 2, Wiley New York,    1998.-   20. Lanckriet et al. Journal of Machine Learning Research, 5:27-72,    2004.-   21. Huang et al. Acoustics, Speech and Signal Processing, IEEE    Transactions on, 27:13-18, February 1979.-   22. Ridler, T. W. and Calvard, S., IEEE Transactions on Systems, Man    and Cybernetics, SMC-8:630-632, 1978.-   23. Dunn, J. C., Journal of Cybernetics, 3:32-57, 1973.-   24. Bezdek, J. C., Pattern Recognition with Fuzzy Objective Function    Algorithms, Plenum Press, 1981.-   25. Chan and Vese, IEEE Transaction on Image Processing, 10:266-277,    2001.-   26. Shapiro, L. G. and Stockman, G. C., Computer Vision, Prentice    Hall, 2001.-   27. Menzies et al. Arch Dermatol, 132(10):1178-1182, 1996.-   28. Argenziano et al. Arch Dermatol, 134:1563-70, 1998.-   29. Elbaum et al. Jour of American Academy of Dermatology,    44(2):207-218, 2001.-   30. StolInitz et al. IEEE Computer Graphics and Applications,    15:76-84, 1995.-   31. Hartigan, J. A. and Wong, M. A., Applied Statistics, 28:100-108,    1979.-   32. Xu et al. Image and Vision Computing, 17:65-74, 1999.-   33. Fawcett, T., Pattern Recognition Letters, 27(8):861-874, 2006.-   36. van de Sande et al. IEEE Transactions on Pattern Analysis and    Machine Intelligence, 32(9):1582-96, 2010.-   35. Ojala et al. IEEE Transactions on Pattern Analysis and Machine    Intelligence, 24:971-987, 2002.-   36. Stanley et al. Skin Research and Technology, 13(1):62-72, 2007.-   37. Chan et al. Journal of Visual Communication and Image    Representation, 11(2):130-141, 2000.-   38. Osher, S. and Sethian, J. A. Journal of Computational Physics,    79(1):12-49, 1988.-   39. Parikh et al. Computer Vision ECCV 2008, pp. 446-459, 2008.-   40. Sanniti di Baja, G. and Svensson, S. In Pattern    Recognition, 2000. Proceedings. 15^(th) International Conference on,    Vol 2:1030-1033, 2000.-   41. Celebi et al. Computerized Medical Imaging and Graphics,    33(2):148-153, 2009.-   42. Silveira et al. IEEE Journal of Selected Topics in Signal    Processing, pages 35-45, 2009.-   43. C. Drummond, C. and Butler, J. R. Emerg Infect Dis, 10.-   44. Fan et al. Visual categorization with bags of keypoints. In    workshop on Statistical Learning in Computer Vision, ECCV, Vol. 22,    2004.-   45. Leung, T. and Malik, J. International Journal of Computer    Vision, 43:29-44, 2001.-   46. June et al. Proc. European Conference on Computer Vision, pgs.    490-503, 2006.-   47. Gool et al. Computer Vision and Image Understanding, 94:3-27,    2004.-   48. Situ et al. Biomedical Imaging: From Nano to Macro, 2011 IEEE    International Symposium, pgs. 109-112, Mar. 30-Apr. 2, 2011.

The present invention is well adapted to attain the ends and advantagesmentioned as well as those that are inherent therein. The particularembodiments disclosed above are illustrative only, as the presentinvention may be modified and practiced in different but equivalentmanners apparent to those skilled in the art having the benefit of theteachings herein. Furthermore, no limitations are intended to thedetails of construction or design herein shown, other than as describedin the claims below. It is therefore evident that the particularillustrative embodiments disclosed above may be altered or modified andall such variations are considered within the scope and spirit of thepresent invention. Also, the terms in the claims have their plain,ordinary meaning unless otherwise explicitly and clearly defined by thepatentee.

What is claimed is:
 1. A portable imaging system, comprising: ahand-held imaging device having a digital camera, a display, a memory, aprocessor and a network connection; and a library of algorithms tangiblystored in the memory and executable by the processor, said algorithmsconfigured for identification of an object of interest present on abody.
 2. The portable imaging system of claim 1, further comprisingalgorithms tangibly stored and processor executable algorithmsconfigured to display the object of interest and results of theclassification thereof.
 3. The portable imaging system of claim 1,wherein the algorithms comprise processor-executable instructions to:segment the imaged object to detect a border of the object; extractfeatures from the segmented object image; and classify the object basedon the extracted features.
 4. The portable imaging system of claim 3,wherein the processor-executable instructions to segment the objectfunction to: determine an initial contour of the imaged object; classifypixels as contained within the initial contour as foreground, ascontained without the initial contour as background or as remainingpixels; and apply a classifier to the remaining pixels forclassification as foreground or background.
 5. The portable imagingsystem of claim 4, wherein the processor-executable instructions toextract features function to: divide the segmented object image intoregions based on saliency values calculated for at least one patchwithin the segmented object; divide the regions into two regions ofhigher or lower intensity based on average intensity values thereof; andextract feature representations from a sampling of patches within theintensity regions based on sampling percentages determined for theregions.
 6. The portable imaging system of claim 5, wherein theprocessor-executable instructions to classify the object function to:input the extracted feature representations into a support vectormachine trained with manually segmented objects; and classify the objectbased on a comparison of the inputted extracted features with those inthe trained support vector machine.
 7. The hand-held imaging system ofclaim 1, wherein the hand-held imaging device is a smart device.
 8. Thehand-held imaging system of claim 1, wherein the body is a human body ora plant body.
 9. The hand-held imaging system of claim 1, wherein theobject of interest is a lesion, an ulcer, or a wound.
 10. A method foridentifying an object of interest present on a body, comprising:acquiring an image of the object of interest on the body via the imagingdevice comprising the portable imaging system of claim 1; processing theacquired object image via the algorithms tangibly stored in the imagingdevice; and identifying the object in the image based on patterns offeatures present in the imaged object, thereby identifying the object ofinterest on the body.
 11. The method of claim 10, further comprising:displaying the results of image processing as each result occurs. 12.The method of claim 10, wherein identifying the object occurs in realtime.
 13. The method of claim 10, wherein the object of interest is amelanoma or a Buruli ulcer.
 14. A digital processor-implemented systemfor classifying an object of interest on an animal or plant body in realtime, comprising: a portable smart device comprising the processor, amemory and a network connection; and modules tangibly stored in thememory comprising: a module for segmentation of an imaged object; amodule for feature extraction within the segmented object image; and amodule for classification of the object based on extracted features. 15.The digital processor-implemented system of claim 14, further comprisinga module tangibly stored in the memory for display of the object ofinterest and results of the classification thereof.
 16. The digitalprocessor-implemented system of claim 14, wherein the segmentationmodule comprises processor executable instructions to: obtain luminanceand color components of the imaged object; classify pixels comprisingthe image as object pixels, if they belong to a common luminance andcolor foreground, as background pixels if they belong to a commonluminance and color background or as remaining pixels; and apply aclassifier to the remaining pixels to classify them as object orforeground.
 17. The digital processor-implemented system of claim 16,wherein the feature extraction module comprises processor executableinstructions to: calculate a saliency value for a plurality of patcheswithin the segmented object and separate the patches into regions basedon the saliency values; calculate an average intensity for the regionsto identify them as a higher or as a lower intensity region; determine asampling percentage for the intensity regions; sample patches within theintensity regions by corresponding sampling percentages; and extract oneor more feature representations for the object.
 18. The digitalprocessor-implemented system of claim 16, wherein the feature extractionmodule comprises processor executable instructions to: read input whitelight image as RGB and the segmentation result of the region; read inputmultispectral images in color channels and transform to gray scale;register multispectral images via maximization of mutual informationwith white light image as reference; extract feature representationswithin the ROI of multispectral images and within white light images;and select one or more relevant features from a pool of the extractedfeatures.
 19. The digital processor-implemented system of claim 16,wherein the feature extraction module comprises processor executableinstructions to: read input white light image as RGB and thesegmentation result of the region; read input multispectral images incolor channels and transform to gray scale; register multispectralimages via maximization of mutual information with white light image asreference; determine V_(mel), V_(blood), and V_(oxy) for each ROI pixelto reconstruct maps of melanin, blood and oxygenating percentage;extract feature representations within the ROI from the reconstructedmaps; and select one or more relevant features from a pool of theextracted features.
 20. The digital processor-implemented system ofclaim 17, wherein the classification module comprises processorexecutable instructions to: train a support vector machine (SVM) withknown manually segmented objects; and classify the object based on theextracted feature representations inputted into the SVM.
 21. Thehand-held imaging system of claim 14, wherein the object of interest isa lesion, an ulcer, a wound, or skin.
 22. A digitalprocessor-implemented method for classifying an object of interest on ananimal or plant body in real time, comprising the processor executablesteps of: digitally imaging the object of interest with the smart devicecomprising the digital processor-implemented system of claim 14;processing the digital image through the system modules, said modulescomprising algorithms configured for: segmenting the image based onsaliency values to identify pixels thereof as comprising the imagedobject or the background of the image to obtain an object boundary;extracting features from regions within the object boundary; andcomparing the extracted features to known object features in a supportvector machine trained on the known features to obtain a classificationof the object; and displaying the processed images and classificationresults on a display comprising the smart device.
 23. The digitalprocessor-implemented method of claim 20, wherein the support vectormachine is trained on features comprising a melanoma or a Buruli ulcer.24. A digital processor-readable medium tangibly storingprocessor-executable instructions to perform the digital processorimplemented method of claim
 20. 25. A computer-readable medium tangiblystoring a library of algorithms to classify an object of interest on ahuman or plant body, said algorithms comprising processor-executableinstructions operable to: obtain luminance and color components of theimaged object; classify pixels comprising the image as object pixels, ifthey belong to a common luminance and color foreground, as backgroundpixels if they belong to a common luminance and color background or asremaining pixels; apply a classifier to the remaining pixels to classifythem as object or foreground; extract one or more featurerepresentations for the object; train a support vector machine (SVM°with known manually segmented objects; and classify the object based onthe extracted feature representations inputted into the SVM.
 26. Thecomputer-readable medium of claim 25, wherein the instructions toextract one or more feature representations for the object comprise:calculate a saliency value for a plurality of patches within thesegmented object and separate the patches into regions based on thesaliency values; calculate an average intensity for the regions toidentify them as a higher or as a lower intensity region; determine asampling percentage for the intensity regions; sample patches within theintensity regions by corresponding sampling percentages; and extract theone or more feature representations for the object.
 27. Thecomputer-readable medium of claim 25, wherein the instructions toextract one or more feature representations for the object comprise:read input white light image as RGB and the segmentation result of theregion; read input multispectral images in color channels and transformto gray scale; register multispectral images via maximization of mutualinformation with white light image as reference; extract featurerepresentations within the ROI of multispectral images and within whitelight images; and select one or more relevant features from a pool ofthe extracted features.
 28. The computer-readable medium of claim 25,wherein the object is the skin, said instructions to extract one or morefeature representations for the object comprising: read input whitelight image as RGB and the segmentation result of the region; read inputmultispectral images in color channels and transform to gray scale;register multispectral images via maximization of mutual informationwith white light image as reference; determine V_(mel), V_(blood), andV_(oxy) for each ROI pixel to reconstruct maps of melanin, blood andoxygenating percentage; extract feature representations within the ROIfrom the reconstructed maps; and select one or more relevant featuresfrom a pool of the extracted features.